Max level shown:
12024-07-26T02:45:01.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 347 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 3d1e580c-07a2-4f73-8110-9aef2adf45d1 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
22024-07-26T02:45:10.738ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
32024-07-26T02:45:10.738ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
42024-07-26T02:45:10.738ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
52024-07-26T02:45:10.738ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
62024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
72024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
92024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
102024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
112024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
122024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
132024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
142024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
152024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
162024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
172024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
182024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
202024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
212024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
222024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
232024-07-26T02:45:10.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
242024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
252024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
262024-07-26T02:45:10.739ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
272024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
282024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
292024-07-26T02:45:10.739ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
302024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
312024-07-26T02:45:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
322024-07-26T02:45:10.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
332024-07-26T02:45:10.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
342024-07-26T02:45:10.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
352024-07-26T02:45:10.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
362024-07-26T02:45:10.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
372024-07-26T02:45:10.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
382024-07-26T02:45:10.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
392024-07-26T02:45:10.741ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12869, tv_nsec: 705065261 } })
402024-07-26T02:45:10.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
412024-07-26T02:45:10.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
422024-07-26T02:45:10.741ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
432024-07-26T02:45:10.742ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
442024-07-26T02:45:10.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
452024-07-26T02:45:10.742ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12869, tv_nsec: 706237558 } })
462024-07-26T02:45:10.742ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
472024-07-26T02:45:10.742ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
482024-07-26T02:45:10.742ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.784865ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
492024-07-26T02:45:10.742ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
502024-07-26T02:45:10.742ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
512024-07-26T02:45:10.742ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
522024-07-26T02:45:10.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
532024-07-26T02:45:10.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.775278ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
542024-07-26T02:45:10.743ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
552024-07-26T02:45:10.744ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
562024-07-26T02:45:10.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
572024-07-26T02:45:10.746ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
582024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 6.503232ms iteration = 14 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
592024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
602024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
612024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
622024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
632024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
642024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
652024-07-26T02:45:10.746ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
662024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
672024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
682024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
692024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
702024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
712024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
722024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
732024-07-26T02:45:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
742024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
752024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
762024-07-26T02:45:10.748ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
772024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
782024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
792024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
802024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
812024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
822024-07-26T02:45:10.748ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12869, tv_nsec: 712785611 } })
832024-07-26T02:45:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
842024-07-26T02:45:10.749ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
852024-07-26T02:45:10.749ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
862024-07-26T02:45:10.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
872024-07-26T02:45:10.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
882024-07-26T02:45:10.750ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
892024-07-26T02:45:10.750ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
902024-07-26T02:45:10.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
912024-07-26T02:45:10.751ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
922024-07-26T02:45:10.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
932024-07-26T02:45:10.752ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
942024-07-26T02:45:10.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
952024-07-26T02:45:10.753ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
962024-07-26T02:45:10.753ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
972024-07-26T02:45:10.753ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 13.825873ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
982024-07-26T02:45:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
992024-07-26T02:45:10.754ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
1002024-07-26T02:45:10.754ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1012024-07-26T02:45:10.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1022024-07-26T02:45:10.760ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
1032024-07-26T02:45:10.760ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 21.103333ms iteration = 15 status = {"physical_disks_added":0}
1042024-07-26T02:45:10.761ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
1052024-07-26T02:45:10.761ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.977223ms iteration = 40 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
1062024-07-26T02:45:10.762ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1072024-07-26T02:45:10.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1082024-07-26T02:45:10.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1092024-07-26T02:45:10.765ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1102024-07-26T02:45:10.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1112024-07-26T02:45:10.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1122024-07-26T02:45:10.767ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1132024-07-26T02:45:10.768ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1142024-07-26T02:45:10.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1152024-07-26T02:45:10.768ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1162024-07-26T02:45:10.768ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1172024-07-26T02:45:10.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1182024-07-26T02:45:10.770ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1192024-07-26T02:45:10.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1202024-07-26T02:45:10.771ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1212024-07-26T02:45:10.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
1222024-07-26T02:45:10.774ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1232024-07-26T02:45:10.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1242024-07-26T02:45:10.775ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 35.97402ms iteration = 14 status = 0
1252024-07-26T02:45:10.887ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
1262024-07-26T02:45:10.887ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1272024-07-26T02:45:10.888ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b1e8e3d2-5577-46fe-a284-f624ca369969", "content-length": "26", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1282024-07-26T02:45:10.888ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1292024-07-26T02:45:10.891ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1302024-07-26T02:45:10.908ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
1312024-07-26T02:45:10.908ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "cd884401-43cd-461e-a6ad-61dd764219f2", "content-length": "372", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1322024-07-26T02:45:10.908ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1332024-07-26T02:45:10.908ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1342024-07-26T02:45:10.910ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1352024-07-26T02:45:10.911ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1362024-07-26T02:45:10.913ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1372024-07-26T02:45:10.913ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1382024-07-26T02:45:10.913ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
1392024-07-26T02:45:10.914ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "46a3b4af-f623-47cc-9b80-1b7581a14ccf", "content-length": "691", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1402024-07-26T02:45:10.914ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
1412024-07-26T02:45:10.914ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
1422024-07-26T02:45:10.914ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
1432024-07-26T02:45:10.915ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
1442024-07-26T02:45:10.915ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 176.171974ms iteration = 14 status = {}
1452024-07-26T02:45:10.915ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "43c75a84-197f-43dd-9590-8dced295be95", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1462024-07-26T02:45:10.917ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 177.73152ms iteration = 14 status = {}
1472024-07-26T02:45:10.978ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
1482024-07-26T02:45:10.978ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1492024-07-26T02:45:10.979ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d19ad841-a640-4127-b892-0509b0461df5", "content-length": "26", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1502024-07-26T02:45:10.979ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1512024-07-26T02:45:10.981ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
1522024-07-26T02:45:10.981ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1532024-07-26T02:45:10.981ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1542024-07-26T02:45:10.981ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d7d52e4e-41ea-4f32-ae21-94acf1d40eb3", "content-length": "26", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1552024-07-26T02:45:10.981ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1562024-07-26T02:45:10.983ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 244.418488ms iteration = 14 status = {}
1572024-07-26T02:45:10.983ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
1582024-07-26T02:45:11.041ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1592024-07-26T02:45:11.042ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0ca5e8e7-5fe9-4d40-b3fa-bed1f6f2d483", "content-length": "1", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1602024-07-26T02:45:11.043ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1612024-07-26T02:45:11.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1622024-07-26T02:45:11.044ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 306.20922ms iteration = 14 status = null
1632024-07-26T02:45:11.104ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1642024-07-26T02:45:11.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1652024-07-26T02:45:11.106ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
1662024-07-26T02:45:11.106ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1672024-07-26T02:45:11.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1682024-07-26T02:45:11.114ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
1692024-07-26T02:45:11.115ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d1fe5f13-ee28-4762-bc8b-8874583798d4", "content-length": "121", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1702024-07-26T02:45:11.115ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
1712024-07-26T02:45:11.115ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
1722024-07-26T02:45:11.116ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
1732024-07-26T02:45:11.116ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
1742024-07-26T02:45:11.116ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
1752024-07-26T02:45:11.116ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
1762024-07-26T02:45:11.116ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
1772024-07-26T02:45:11.116ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1782024-07-26T02:45:11.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1792024-07-26T02:45:11.117ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
1802024-07-26T02:45:11.117ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9861ef04-8269-4b6f-95ae-f0af994a0672", "content-length": "2", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1812024-07-26T02:45:11.117ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
1822024-07-26T02:45:11.117ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e10fb557-e81c-4c85-9e1d-cf63c9f331fb", "content-length": "52", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1832024-07-26T02:45:11.117ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
1842024-07-26T02:45:11.117ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
1852024-07-26T02:45:11.167ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "0644b5e5-ce87-4576-9757-e14afaa78e86", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1862024-07-26T02:45:11.167ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
1872024-07-26T02:45:11.167ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
1882024-07-26T02:45:11.168ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "59dde202-1215-4270-9969-eea04e87e7e7", "content-length": "494", "date": "Fri, 26 Jul 2024 02:45:10 GMT"} })
1892024-07-26T02:45:11.168ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1902024-07-26T02:45:11.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1912024-07-26T02:45:11.171ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1922024-07-26T02:45:11.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1932024-07-26T02:45:11.174ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1942024-07-26T02:45:11.174ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1952024-07-26T02:45:11.175ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1962024-07-26T02:45:11.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1972024-07-26T02:45:11.182ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 442.798189ms iteration = 14 status = {}
1982024-07-26T02:45:11.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 336 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 73971fc4-d7a2-4666-b3b9-6e2ec530b90b response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
1992024-07-26T02:45:20.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
2002024-07-26T02:45:20.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
2012024-07-26T02:45:20.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2022024-07-26T02:45:20.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
2032024-07-26T02:45:20.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2042024-07-26T02:45:20.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2052024-07-26T02:45:20.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2062024-07-26T02:45:20.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2072024-07-26T02:45:20.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2082024-07-26T02:45:20.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2092024-07-26T02:45:20.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
2102024-07-26T02:45:20.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.007633ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2112024-07-26T02:45:20.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2122024-07-26T02:45:20.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2132024-07-26T02:45:20.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2142024-07-26T02:45:20.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2152024-07-26T02:45:20.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2162024-07-26T02:45:20.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2172024-07-26T02:45:20.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
2182024-07-26T02:45:20.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2192024-07-26T02:45:20.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2202024-07-26T02:45:20.755ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
2212024-07-26T02:45:20.755ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.473805ms iteration = 41 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
2222024-07-26T02:45:21.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 842 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = a3428f21-fc06-4cf9-8beb-9c200e6fb3e3 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
2232024-07-26T02:45:24.738ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::1:0]:44254
2242024-07-26T02:45:24.739ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:44254 req_id = f5daecdf-ff2a-42f0-a510-a0919a163c33 roles = RoleSet { roles: {} } uri = /metrics/producers
2252024-07-26T02:45:24.739ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:44254 req_id = f5daecdf-ff2a-42f0-a510-a0919a163c33 resource = Database result = Ok(()) uri = /metrics/producers
2262024-07-26T02:45:24.771ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:338 id = 8752c2d7-36d8-49ed-860d-ec196a32b72c
2272024-07-26T02:45:24.771ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:44254 req_id = f5daecdf-ff2a-42f0-a510-a0919a163c33 roles = RoleSet { roles: {} } uri = /metrics/producers
2282024-07-26T02:45:24.771ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:44254 req_id = f5daecdf-ff2a-42f0-a510-a0919a163c33 resource = Database result = Ok(()) uri = /metrics/producers
2292024-07-26T02:45:24.806ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = 8752c2d7-36d8-49ed-860d-ec196a32b72c uri = http://[fd00:1122:3344:101::d]:12223/producers
2302024-07-26T02:45:24.806ZDEBGnexus: client response oximeter-collector = 8752c2d7-36d8-49ed-860d-ec196a32b72c result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6535460d-d66b-4032-9323-a7d037b06192", "date": "Fri, 26 Jul 2024 02:45:24 GMT"} })
2312024-07-26T02:45:24.806ZINFOnexus: assigned collector to new producer collector_id = 8752c2d7-36d8-49ed-860d-ec196a32b72c file = nexus/src/app/oximeter.rs:125 producer_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
2322024-07-26T02:45:24.808ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 69115 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:44254 req_id = f5daecdf-ff2a-42f0-a510-a0919a163c33 response_code = 201 uri = /metrics/producers
2332024-07-26T02:45:30.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
2342024-07-26T02:45:30.740ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
2352024-07-26T02:45:30.740ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2362024-07-26T02:45:30.740ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
2372024-07-26T02:45:30.740ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2382024-07-26T02:45:30.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2392024-07-26T02:45:30.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2402024-07-26T02:45:30.742ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2412024-07-26T02:45:30.742ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2422024-07-26T02:45:30.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2432024-07-26T02:45:30.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
2442024-07-26T02:45:30.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.84645ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2452024-07-26T02:45:30.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2462024-07-26T02:45:30.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2472024-07-26T02:45:30.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2482024-07-26T02:45:30.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2492024-07-26T02:45:30.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2502024-07-26T02:45:30.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2512024-07-26T02:45:30.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
2522024-07-26T02:45:30.751ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2532024-07-26T02:45:30.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2542024-07-26T02:45:30.757ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
2552024-07-26T02:45:30.757ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.95062ms iteration = 42 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
2562024-07-26T02:45:31.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 572 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 24e057c3-9254-49aa-8db7-107e9206aed9 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
2572024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
2582024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2592024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
2602024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
2612024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2622024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
2632024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
2642024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
2652024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
2662024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
2672024-07-26T02:45:40.738ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2024-07-26 02:35:40.738776481 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
2682024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
2692024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
2702024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
2712024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
2722024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2732024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
2742024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2752024-07-26T02:45:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
2762024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2772024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2782024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
2792024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2802024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2812024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
2822024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
2832024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
2842024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
2852024-07-26T02:45:40.739ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
2862024-07-26T02:45:40.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
2872024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
2882024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2892024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
2902024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
2912024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2922024-07-26T02:45:40.739ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:69 reason = blueprint disabled target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a
2932024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2942024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2952024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 7.481\u{b5}s iteration = 9 status = {"error":"blueprint disabled","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a"}
2962024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
2972024-07-26T02:45:40.739ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
2982024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
2992024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3002024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
3012024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3022024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
3032024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
3042024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3052024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3062024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3072024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
3082024-07-26T02:45:40.739ZINFOnexus (BackgroundTasks): Abandoned VMM reaper running background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:76
3092024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
3102024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
3112024-07-26T02:45:40.739ZINFOnexus (BackgroundTasks): lookup region port task started background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:56
3122024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
3132024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
3142024-07-26T02:45:40.739ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
3152024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
3162024-07-26T02:45:40.739ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53979846-b429-4bae-9968-d96ff860d07e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 12899, tv_nsec: 703604959 } })
3172024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3182024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3192024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3202024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
3212024-07-26T02:45:40.739ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("25724e7e-f0f3-413d-ab41-b4a33973050b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4e64b0cb-e3e2-4959-adf4-d5cd70f25b40.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 12899, tv_nsec: 703820525 } })
3222024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
3232024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3242024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
3252024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
3262024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3272024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3282024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
3292024-07-26T02:45:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3302024-07-26T02:45:40.740ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3312024-07-26T02:45:40.740ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3322024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3332024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3342024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3352024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
3362024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 2.617161ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
3372024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3382024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3392024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3402024-07-26T02:45:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3412024-07-26T02:45:40.742ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
3422024-07-26T02:45:40.742ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 3.603844ms iteration = 8 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
3432024-07-26T02:45:40.742ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
3442024-07-26T02:45:40.742ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.192285ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
3452024-07-26T02:45:40.742ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3462024-07-26T02:45:40.742ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3472024-07-26T02:45:40.743ZINFOnexus (BackgroundTasks): lookup region port task done background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:150
3482024-07-26T02:45:40.743ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 4.079715ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
3492024-07-26T02:45:40.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
3502024-07-26T02:45:40.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.360252ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3512024-07-26T02:45:40.744ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3522024-07-26T02:45:40.744ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12899, tv_nsec: 708428232 } })
3532024-07-26T02:45:40.744ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12899, tv_nsec: 708522005 } })
3542024-07-26T02:45:40.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
3552024-07-26T02:45:40.744ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3562024-07-26T02:45:40.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3572024-07-26T02:45:40.744ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2024-07-26 02:35:40.738776481 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
3582024-07-26T02:45:40.744ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.191885ms iteration = 8 status = {"expiration":"2024-07-26T02:35:40.738776481Z","pruned":[]}
3592024-07-26T02:45:40.745ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3602024-07-26T02:45:40.745ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3612024-07-26T02:45:40.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3622024-07-26T02:45:40.745ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
3632024-07-26T02:45:40.745ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
3642024-07-26T02:45:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3652024-07-26T02:45:40.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3662024-07-26T02:45:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3672024-07-26T02:45:40.746ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3682024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3692024-07-26T02:45:40.747ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
3702024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3712024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3722024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3732024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3742024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3752024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
3762024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3772024-07-26T02:45:40.747ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3782024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3792024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3802024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3812024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
3822024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
3832024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
3842024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3852024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
3862024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
3872024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
3882024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3892024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3902024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3912024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3922024-07-26T02:45:40.748ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3932024-07-26T02:45:40.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3942024-07-26T02:45:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
3952024-07-26T02:45:40.749ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3962024-07-26T02:45:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
3972024-07-26T02:45:40.749ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3982024-07-26T02:45:40.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3992024-07-26T02:45:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
4002024-07-26T02:45:40.752ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4012024-07-26T02:45:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
4022024-07-26T02:45:40.752ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4032024-07-26T02:45:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4042024-07-26T02:45:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4052024-07-26T02:45:40.753ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4062024-07-26T02:45:40.753ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4072024-07-26T02:45:40.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4082024-07-26T02:45:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
4092024-07-26T02:45:40.755ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4102024-07-26T02:45:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4112024-07-26T02:45:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
4122024-07-26T02:45:40.755ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4132024-07-26T02:45:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4142024-07-26T02:45:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4152024-07-26T02:45:40.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4162024-07-26T02:45:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4172024-07-26T02:45:40.757ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4182024-07-26T02:45:40.760ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4192024-07-26T02:45:40.760ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4202024-07-26T02:45:40.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4212024-07-26T02:45:40.762ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4222024-07-26T02:45:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4232024-07-26T02:45:40.765ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4242024-07-26T02:45:40.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4252024-07-26T02:45:40.766ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12899, tv_nsec: 730110358 } })
4262024-07-26T02:45:40.766ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
4272024-07-26T02:45:40.766ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
4282024-07-26T02:45:40.766ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 27.185045ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
4292024-07-26T02:45:40.767ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4302024-07-26T02:45:40.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4312024-07-26T02:45:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4322024-07-26T02:45:40.769ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4332024-07-26T02:45:40.769ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4342024-07-26T02:45:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4352024-07-26T02:45:40.769ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4362024-07-26T02:45:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4372024-07-26T02:45:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4382024-07-26T02:45:40.770ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4392024-07-26T02:45:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4402024-07-26T02:45:40.772ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
4412024-07-26T02:45:40.772ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 0 version = 2
4422024-07-26T02:45:40.772ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4432024-07-26T02:45:40.772ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4442024-07-26T02:45:40.772ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4452024-07-26T02:45:40.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4462024-07-26T02:45:40.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4472024-07-26T02:45:40.773ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
4482024-07-26T02:45:40.773ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
4492024-07-26T02:45:40.773ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4502024-07-26T02:45:40.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4512024-07-26T02:45:40.774ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4522024-07-26T02:45:40.775ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4532024-07-26T02:45:40.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4542024-07-26T02:45:40.778ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4552024-07-26T02:45:40.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4562024-07-26T02:45:40.778ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4572024-07-26T02:45:40.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4582024-07-26T02:45:40.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4592024-07-26T02:45:40.779ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4602024-07-26T02:45:40.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4612024-07-26T02:45:40.780ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4622024-07-26T02:45:40.780ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 1 version = 2
4632024-07-26T02:45:40.781ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
4642024-07-26T02:45:40.781ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-07-26 02:38:55.456872 UTC dns_group = external generation = 2
4652024-07-26T02:45:40.781ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 42.148115ms iteration = 8 status = {"generation":2}
4662024-07-26T02:45:40.783ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
4672024-07-26T02:45:40.783ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4682024-07-26T02:45:40.783ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
4692024-07-26T02:45:40.783ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
4702024-07-26T02:45:40.783ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-07-26 02:38:54.868472 UTC dns_group = internal generation = 1
4712024-07-26T02:45:40.783ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 44.781216ms iteration = 8 status = {"generation":1}
4722024-07-26T02:45:40.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4732024-07-26T02:45:40.784ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4742024-07-26T02:45:40.787ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
4752024-07-26T02:45:40.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4762024-07-26T02:45:40.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4772024-07-26T02:45:40.787ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4782024-07-26T02:45:40.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4792024-07-26T02:45:40.789ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
4802024-07-26T02:45:40.789ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 49.967207ms iteration = 15 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
4812024-07-26T02:45:40.789ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4822024-07-26T02:45:40.789ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-07-26T02:38:40.312512Z, time_modified: 2024-07-26T02:38:40.312512Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
4832024-07-26T02:45:40.789ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 50.737785ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]}},"default_endpoint":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
4842024-07-26T02:45:40.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4852024-07-26T02:45:40.790ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
4862024-07-26T02:45:40.790ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.292258ms iteration = 43 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
4872024-07-26T02:45:40.791ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4882024-07-26T02:45:40.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4892024-07-26T02:45:40.792ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4902024-07-26T02:45:40.796ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0
4912024-07-26T02:45:40.796ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:189 found = 0 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0
4922024-07-26T02:45:40.796ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 57.196555ms iteration = 8 status = {"error":null,"error_count":0,"found":0,"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0}
4932024-07-26T02:45:40.797ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
4942024-07-26T02:45:40.797ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4952024-07-26T02:45:40.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4962024-07-26T02:45:40.798ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 59.403257ms iteration = 15 status = 0
4972024-07-26T02:45:40.804ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
4982024-07-26T02:45:40.804ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
4992024-07-26T02:45:40.804ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 64.744601ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
5002024-07-26T02:45:40.804ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
5012024-07-26T02:45:40.804ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 65.417678ms iteration = 16 status = {"physical_disks_added":0}
5022024-07-26T02:45:40.887ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
5032024-07-26T02:45:40.906ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
5042024-07-26T02:45:40.949ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
5052024-07-26T02:45:40.949ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f8f3a9ef-d01f-40d2-82ad-955914f024c2", "content-length": "372", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5062024-07-26T02:45:40.949ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5072024-07-26T02:45:40.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5082024-07-26T02:45:40.952ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5092024-07-26T02:45:40.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5102024-07-26T02:45:40.954ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5112024-07-26T02:45:40.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5122024-07-26T02:45:40.956ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
5132024-07-26T02:45:40.958ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a13c8226-d2f7-463e-8fa6-295d70b69168", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5142024-07-26T02:45:40.958ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
5152024-07-26T02:45:40.958ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5162024-07-26T02:45:40.958ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d71b4831-4a35-4bbe-9596-ef3999c6a3e4", "content-length": "26", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5172024-07-26T02:45:40.958ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5182024-07-26T02:45:40.961ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 222.072105ms iteration = 15 status = {}
5192024-07-26T02:45:40.961ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
5202024-07-26T02:45:40.961ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5212024-07-26T02:45:40.961ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5222024-07-26T02:45:40.961ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "82838f0b-a3d1-4536-b66b-4539ba99d360", "content-length": "26", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5232024-07-26T02:45:40.961ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5242024-07-26T02:45:40.963ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
5252024-07-26T02:45:40.963ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5262024-07-26T02:45:40.963ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
5272024-07-26T02:45:40.963ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f8837c73-05ae-4a4b-8fa0-ddad99ba4efa", "content-length": "26", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5282024-07-26T02:45:40.963ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5292024-07-26T02:45:40.965ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5302024-07-26T02:45:41.004ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
5312024-07-26T02:45:41.006ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9fbe8732-9e54-4175-ba53-2c7507e3a952", "content-length": "691", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5322024-07-26T02:45:41.006ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
5332024-07-26T02:45:41.006ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
5342024-07-26T02:45:41.006ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
5352024-07-26T02:45:41.007ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 268.462309ms iteration = 15 status = {}
5362024-07-26T02:45:41.039ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
5372024-07-26T02:45:41.039ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a14a5ec1-600e-4c2f-9c25-c75e42e4b069", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5382024-07-26T02:45:41.039ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5392024-07-26T02:45:41.041ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "1fdc2e40-78da-43d9-9036-bef4d5810bb5", "date": "Fri, 26 Jul 2024 02:45:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5402024-07-26T02:45:41.041ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5412024-07-26T02:45:41.043ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
5422024-07-26T02:45:41.043ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 304.346158ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
5432024-07-26T02:45:41.066ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
5442024-07-26T02:45:41.097ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
5452024-07-26T02:45:41.097ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0b078f4e-b57f-4a16-93ed-c588ec186b4f", "content-length": "2", "date": "Fri, 26 Jul 2024 02:45:41 GMT"} })
5462024-07-26T02:45:41.099ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 360.701672ms iteration = 15 status = {}
5472024-07-26T02:45:41.100ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5482024-07-26T02:45:41.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5492024-07-26T02:45:41.102ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 363.60293ms iteration = 15 status = null
5502024-07-26T02:45:41.125ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
5512024-07-26T02:45:41.125ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "55997935-e886-480b-934a-91677e6d7f9c", "date": "Fri, 26 Jul 2024 02:45:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
5522024-07-26T02:45:41.125ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
5532024-07-26T02:45:41.128ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4e16b540-cf29-4091-84bf-63acd37146d8", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
5542024-07-26T02:45:41.128ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
5552024-07-26T02:45:41.129ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5f1f3f50-6103-4c0d-86c2-b555ee3e6db8", "date": "Fri, 26 Jul 2024 02:45:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
5562024-07-26T02:45:41.129ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
5572024-07-26T02:45:41.131ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
5582024-07-26T02:45:41.131ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 392.959466ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
5592024-07-26T02:45:41.164ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5602024-07-26T02:45:41.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5612024-07-26T02:45:41.166ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
5622024-07-26T02:45:41.166ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5632024-07-26T02:45:41.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5642024-07-26T02:45:41.174ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
5652024-07-26T02:45:41.175ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e952b308-e4e5-4a49-bf76-9695b3ef0981", "content-length": "121", "date": "Fri, 26 Jul 2024 02:45:41 GMT"} })
5662024-07-26T02:45:41.175ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
5672024-07-26T02:45:41.175ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
5682024-07-26T02:45:41.176ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
5692024-07-26T02:45:41.176ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
5702024-07-26T02:45:41.176ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
5712024-07-26T02:45:41.176ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
5722024-07-26T02:45:41.176ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
5732024-07-26T02:45:41.176ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5742024-07-26T02:45:41.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5752024-07-26T02:45:41.177ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
5762024-07-26T02:45:41.178ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d95363b2-f8ff-466b-969d-efe30d8aa62a", "content-length": "2", "date": "Fri, 26 Jul 2024 02:45:41 GMT"} })
5772024-07-26T02:45:41.178ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
5782024-07-26T02:45:41.178ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7463f67b-2b4f-4aa2-a4ac-15cfcc451345", "content-length": "52", "date": "Fri, 26 Jul 2024 02:45:41 GMT"} })
5792024-07-26T02:45:41.178ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
5802024-07-26T02:45:41.178ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
5812024-07-26T02:45:41.228ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "ca527727-3fd9-4264-9fcb-a2fea8543f7f", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5822024-07-26T02:45:41.228ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
5832024-07-26T02:45:41.228ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
5842024-07-26T02:45:41.229ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "32034632-52ac-49ac-819c-54213decea09", "content-length": "494", "date": "Fri, 26 Jul 2024 02:45:40 GMT"} })
5852024-07-26T02:45:41.229ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5862024-07-26T02:45:41.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5872024-07-26T02:45:41.232ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5882024-07-26T02:45:41.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5892024-07-26T02:45:41.234ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5902024-07-26T02:45:41.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5912024-07-26T02:45:41.236ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5922024-07-26T02:45:41.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5932024-07-26T02:45:41.242ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 503.674511ms iteration = 15 status = {}
5942024-07-26T02:45:41.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 408 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = fa4ff4e8-39cc-4c50-86db-c3c744bec4d2 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
5952024-07-26T02:45:50.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
5962024-07-26T02:45:50.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
5972024-07-26T02:45:50.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
5982024-07-26T02:45:50.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5992024-07-26T02:45:50.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6002024-07-26T02:45:50.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6012024-07-26T02:45:50.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6022024-07-26T02:45:50.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6032024-07-26T02:45:50.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6042024-07-26T02:45:50.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6052024-07-26T02:45:50.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
6062024-07-26T02:45:50.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.485941ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6072024-07-26T02:45:50.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6082024-07-26T02:45:50.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6092024-07-26T02:45:50.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6102024-07-26T02:45:50.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6112024-07-26T02:45:50.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6122024-07-26T02:45:50.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6132024-07-26T02:45:50.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
6142024-07-26T02:45:50.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6152024-07-26T02:45:50.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6162024-07-26T02:45:50.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
6172024-07-26T02:45:50.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.047408ms iteration = 44 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
6182024-07-26T02:45:51.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 617 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 999f3513-e93a-4788-88a0-f662151fb0ea response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
6192024-07-26T02:46:00.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
6202024-07-26T02:46:00.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
6212024-07-26T02:46:00.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
6222024-07-26T02:46:00.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6232024-07-26T02:46:00.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6242024-07-26T02:46:00.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6252024-07-26T02:46:00.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6262024-07-26T02:46:00.742ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6272024-07-26T02:46:00.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6282024-07-26T02:46:00.742ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6292024-07-26T02:46:00.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
6302024-07-26T02:46:00.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.329561ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6312024-07-26T02:46:00.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6322024-07-26T02:46:00.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6332024-07-26T02:46:00.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6342024-07-26T02:46:00.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6352024-07-26T02:46:00.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6362024-07-26T02:46:00.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6372024-07-26T02:46:00.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
6382024-07-26T02:46:00.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6392024-07-26T02:46:00.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6402024-07-26T02:46:00.757ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
6412024-07-26T02:46:00.757ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.156675ms iteration = 45 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
6422024-07-26T02:46:01.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 549 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 3e813e34-0e9b-4f29-b79f-709886ad0024 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
6432024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
6442024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
6452024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6462024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6472024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
6482024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6492024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
6502024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
6512024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
6522024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6532024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
6542024-07-26T02:46:10.738ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
6552024-07-26T02:46:10.738ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
6562024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
6572024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6582024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
6592024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6602024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
6612024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6622024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
6632024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
6642024-07-26T02:46:10.739ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
6652024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6662024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
6672024-07-26T02:46:10.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
6682024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6692024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6702024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6712024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6722024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
6732024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6742024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
6752024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6762024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6772024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6782024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6792024-07-26T02:46:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6802024-07-26T02:46:10.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6812024-07-26T02:46:10.741ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6822024-07-26T02:46:10.741ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6832024-07-26T02:46:10.742ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6842024-07-26T02:46:10.742ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
6852024-07-26T02:46:10.742ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.42744ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
6862024-07-26T02:46:10.743ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12929, tv_nsec: 707418410 } })
6872024-07-26T02:46:10.743ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6882024-07-26T02:46:10.744ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
6892024-07-26T02:46:10.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
6902024-07-26T02:46:10.744ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6912024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6922024-07-26T02:46:10.745ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12929, tv_nsec: 709428846 } })
6932024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
6942024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6952024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6962024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6972024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
6982024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6992024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7002024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
7012024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7022024-07-26T02:46:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7032024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7042024-07-26T02:46:10.746ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
7052024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7062024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
7072024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7082024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7092024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7102024-07-26T02:46:10.746ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
7112024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7122024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7132024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7142024-07-26T02:46:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
7152024-07-26T02:46:10.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7162024-07-26T02:46:10.747ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
7172024-07-26T02:46:10.747ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 8.471298ms iteration = 16 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
7182024-07-26T02:46:10.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7192024-07-26T02:46:10.747ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12929, tv_nsec: 712153680 } })
7202024-07-26T02:46:10.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7212024-07-26T02:46:10.748ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
7222024-07-26T02:46:10.748ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
7232024-07-26T02:46:10.748ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 9.524133ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7242024-07-26T02:46:10.749ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
7252024-07-26T02:46:10.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7262024-07-26T02:46:10.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7272024-07-26T02:46:10.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
7282024-07-26T02:46:10.751ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7292024-07-26T02:46:10.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7302024-07-26T02:46:10.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
7312024-07-26T02:46:10.751ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7322024-07-26T02:46:10.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7332024-07-26T02:46:10.753ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7342024-07-26T02:46:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7352024-07-26T02:46:10.754ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7362024-07-26T02:46:10.754ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7372024-07-26T02:46:10.754ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
7382024-07-26T02:46:10.754ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 15.706647ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
7392024-07-26T02:46:10.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7402024-07-26T02:46:10.758ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
7412024-07-26T02:46:10.758ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7422024-07-26T02:46:10.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7432024-07-26T02:46:10.759ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
7442024-07-26T02:46:10.759ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 20.830247ms iteration = 17 status = {"physical_disks_added":0}
7452024-07-26T02:46:10.760ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7462024-07-26T02:46:10.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7472024-07-26T02:46:10.762ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7482024-07-26T02:46:10.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7492024-07-26T02:46:10.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7502024-07-26T02:46:10.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7512024-07-26T02:46:10.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7522024-07-26T02:46:10.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7532024-07-26T02:46:10.765ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
7542024-07-26T02:46:10.765ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.622021ms iteration = 46 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
7552024-07-26T02:46:10.766ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
7562024-07-26T02:46:10.766ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7572024-07-26T02:46:10.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7582024-07-26T02:46:10.766ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7592024-07-26T02:46:10.766ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7602024-07-26T02:46:10.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7612024-07-26T02:46:10.767ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 28.665259ms iteration = 16 status = 0
7622024-07-26T02:46:10.767ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7632024-07-26T02:46:10.769ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7642024-07-26T02:46:10.769ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7652024-07-26T02:46:10.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7662024-07-26T02:46:10.859ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
7672024-07-26T02:46:10.859ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7682024-07-26T02:46:10.859ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b07cd3e0-dacf-4c8c-ac85-3bf1aeee8d8b", "content-length": "26", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
7692024-07-26T02:46:10.859ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7702024-07-26T02:46:10.860ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7712024-07-26T02:46:10.866ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
7722024-07-26T02:46:10.866ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7732024-07-26T02:46:10.867ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b7d141ff-4a1c-4969-8628-bf9b1469a0b8", "content-length": "26", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
7742024-07-26T02:46:10.867ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7752024-07-26T02:46:10.869ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7762024-07-26T02:46:10.872ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7772024-07-26T02:46:10.872ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
7782024-07-26T02:46:10.872ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7792024-07-26T02:46:10.872ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e61ec9cf-e78b-42c5-957f-2b754398079e", "content-length": "372", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
7802024-07-26T02:46:10.872ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7812024-07-26T02:46:10.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7822024-07-26T02:46:10.873ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3102b57d-be4c-455e-8799-7e15c893b223", "content-length": "26", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
7832024-07-26T02:46:10.873ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7842024-07-26T02:46:10.874ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
7852024-07-26T02:46:10.875ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7862024-07-26T02:46:10.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7872024-07-26T02:46:10.877ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7882024-07-26T02:46:10.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7892024-07-26T02:46:10.879ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7902024-07-26T02:46:10.879ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "89658dd9-ec94-4bc0-94c4-75aaffd78b45", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
7912024-07-26T02:46:10.882ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 143.139483ms iteration = 16 status = {}
7922024-07-26T02:46:10.898ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
7932024-07-26T02:46:10.898ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "eafca982-b933-408c-a572-e9e3f5edeebf", "content-length": "691", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
7942024-07-26T02:46:10.898ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
7952024-07-26T02:46:10.898ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
7962024-07-26T02:46:10.898ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
7972024-07-26T02:46:10.901ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 162.053254ms iteration = 16 status = {}
7982024-07-26T02:46:11.061ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 322.668105ms iteration = 16 status = {}
7992024-07-26T02:46:11.065ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
8002024-07-26T02:46:11.066ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "eee7ea4a-7b90-4801-93c4-6a5cda1659ab", "content-length": "2", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
8012024-07-26T02:46:11.068ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8022024-07-26T02:46:11.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8032024-07-26T02:46:11.069ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 331.211605ms iteration = 16 status = null
8042024-07-26T02:46:11.090ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8052024-07-26T02:46:11.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8062024-07-26T02:46:11.092ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
8072024-07-26T02:46:11.092ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8082024-07-26T02:46:11.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8092024-07-26T02:46:11.101ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
8102024-07-26T02:46:11.101ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd53356-774d-4200-bc20-552d419acd67", "content-length": "121", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
8112024-07-26T02:46:11.101ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
8122024-07-26T02:46:11.101ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
8132024-07-26T02:46:11.101ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
8142024-07-26T02:46:11.101ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
8152024-07-26T02:46:11.101ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
8162024-07-26T02:46:11.101ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
8172024-07-26T02:46:11.101ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
8182024-07-26T02:46:11.101ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8192024-07-26T02:46:11.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8202024-07-26T02:46:11.103ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
8212024-07-26T02:46:11.103ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "6cdf882d-314c-487e-9807-0cd919401eee", "content-length": "2", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
8222024-07-26T02:46:11.103ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
8232024-07-26T02:46:11.103ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5badfd3d-bd17-4d9b-8eb5-c876c795a5f9", "content-length": "52", "date": "Fri, 26 Jul 2024 02:46:10 GMT"} })
8242024-07-26T02:46:11.103ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
8252024-07-26T02:46:11.103ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
8262024-07-26T02:46:11.143ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "642b2a86-527c-4c2c-a7b8-2ffe7aff7dd4", "date": "Fri, 26 Jul 2024 02:46:11 GMT"} })
8272024-07-26T02:46:11.143ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
8282024-07-26T02:46:11.143ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
8292024-07-26T02:46:11.143ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5d0f1015-3557-45bf-9949-4f3f3b8f5f9f", "content-length": "494", "date": "Fri, 26 Jul 2024 02:46:11 GMT"} })
8302024-07-26T02:46:11.143ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8312024-07-26T02:46:11.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8322024-07-26T02:46:11.146ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8332024-07-26T02:46:11.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8342024-07-26T02:46:11.149ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8352024-07-26T02:46:11.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8362024-07-26T02:46:11.150ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8372024-07-26T02:46:11.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8382024-07-26T02:46:11.157ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 418.718527ms iteration = 16 status = {}
8392024-07-26T02:46:11.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 498 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = befc868b-7e79-411e-a408-220acc070c5f response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
8402024-07-26T02:46:20.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
8412024-07-26T02:46:20.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8422024-07-26T02:46:20.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
8432024-07-26T02:46:20.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
8442024-07-26T02:46:20.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8452024-07-26T02:46:20.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8462024-07-26T02:46:20.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8472024-07-26T02:46:20.742ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8482024-07-26T02:46:20.742ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8492024-07-26T02:46:20.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8502024-07-26T02:46:20.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
8512024-07-26T02:46:20.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.699947ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8522024-07-26T02:46:20.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8532024-07-26T02:46:20.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8542024-07-26T02:46:20.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8552024-07-26T02:46:20.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8562024-07-26T02:46:20.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8572024-07-26T02:46:20.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8582024-07-26T02:46:20.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
8592024-07-26T02:46:20.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8602024-07-26T02:46:20.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8612024-07-26T02:46:20.757ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
8622024-07-26T02:46:20.757ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.496468ms iteration = 47 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
8632024-07-26T02:46:21.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 356 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 8e0e4d10-dc99-4079-9b6d-77d4ca504181 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
8642024-07-26T02:46:30.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
8652024-07-26T02:46:30.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
8662024-07-26T02:46:30.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8672024-07-26T02:46:30.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
8682024-07-26T02:46:30.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8692024-07-26T02:46:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8702024-07-26T02:46:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8712024-07-26T02:46:30.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8722024-07-26T02:46:30.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8732024-07-26T02:46:30.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8742024-07-26T02:46:30.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
8752024-07-26T02:46:30.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.971603ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8762024-07-26T02:46:30.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8772024-07-26T02:46:30.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8782024-07-26T02:46:30.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8792024-07-26T02:46:30.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8802024-07-26T02:46:30.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8812024-07-26T02:46:30.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8822024-07-26T02:46:30.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
8832024-07-26T02:46:30.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8842024-07-26T02:46:30.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8852024-07-26T02:46:30.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
8862024-07-26T02:46:30.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.11155ms iteration = 48 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
8872024-07-26T02:46:31.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 536 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 4d327a3e-501c-4d04-be68-cb6e0d3085a7 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
8882024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
8892024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8902024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
8912024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8922024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
8932024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
8942024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
8952024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
8962024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
8972024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
8982024-07-26T02:46:40.738ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2024-07-26 02:36:40.738701369 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
8992024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
9002024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
9012024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
9022024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9032024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
9042024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
9052024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9062024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
9072024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
9082024-07-26T02:46:40.738ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
9092024-07-26T02:46:40.739ZINFOnexus (BackgroundTasks): Abandoned VMM reaper running background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:76
9102024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9112024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
9122024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
9132024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
9142024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9152024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
9162024-07-26T02:46:40.739ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
9172024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9182024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
9192024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
9202024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
9212024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9222024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
9232024-07-26T02:46:40.739ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
9242024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9252024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9262024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
9272024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9282024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
9292024-07-26T02:46:40.739ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
9302024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
9312024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
9322024-07-26T02:46:40.739ZINFOnexus (BackgroundTasks): lookup region port task started background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:56
9332024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
9342024-07-26T02:46:40.739ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:69 reason = blueprint disabled target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a
9352024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
9362024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 21.121\u{b5}s iteration = 10 status = {"error":"blueprint disabled","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a"}
9372024-07-26T02:46:40.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
9382024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9392024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
9402024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9412024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9422024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9432024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
9442024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9452024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
9462024-07-26T02:46:40.739ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53979846-b429-4bae-9968-d96ff860d07e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 12959, tv_nsec: 704029203 } })
9472024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9482024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9492024-07-26T02:46:40.739ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("25724e7e-f0f3-413d-ab41-b4a33973050b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4e64b0cb-e3e2-4959-adf4-d5cd70f25b40.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 12959, tv_nsec: 704106525 } })
9502024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9512024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
9522024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9532024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
9542024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9552024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9562024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
9572024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9582024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9592024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9602024-07-26T02:46:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9612024-07-26T02:46:40.740ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9622024-07-26T02:46:40.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9632024-07-26T02:46:40.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9642024-07-26T02:46:40.741ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
9652024-07-26T02:46:40.741ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 3.068811ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
9662024-07-26T02:46:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9672024-07-26T02:46:40.742ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
9682024-07-26T02:46:40.742ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 3.837929ms iteration = 9 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
9692024-07-26T02:46:40.742ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9702024-07-26T02:46:40.743ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
9712024-07-26T02:46:40.743ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.406673ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9722024-07-26T02:46:40.743ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12959, tv_nsec: 708173340 } })
9732024-07-26T02:46:40.744ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9742024-07-26T02:46:40.744ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2024-07-26 02:36:40.738701369 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
9752024-07-26T02:46:40.744ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.671082ms iteration = 9 status = {"expiration":"2024-07-26T02:36:40.738701369Z","pruned":[]}
9762024-07-26T02:46:40.744ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9772024-07-26T02:46:40.745ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
9782024-07-26T02:46:40.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9792024-07-26T02:46:40.745ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9802024-07-26T02:46:40.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9812024-07-26T02:46:40.745ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9822024-07-26T02:46:40.745ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12959, tv_nsec: 709820859 } })
9832024-07-26T02:46:40.745ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9842024-07-26T02:46:40.745ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
9852024-07-26T02:46:40.746ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9862024-07-26T02:46:40.747ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9872024-07-26T02:46:40.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9882024-07-26T02:46:40.747ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9892024-07-26T02:46:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9902024-07-26T02:46:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9912024-07-26T02:46:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
9922024-07-26T02:46:40.748ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
9932024-07-26T02:46:40.748ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
9942024-07-26T02:46:40.748ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.85686ms iteration = 17 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
9952024-07-26T02:46:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
9962024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9972024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9982024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9992024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10002024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10012024-07-26T02:46:40.749ZINFOnexus (BackgroundTasks): lookup region port task done background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:150
10022024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.121016ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
10032024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10042024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10052024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10062024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10072024-07-26T02:46:40.749ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10082024-07-26T02:46:40.750ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10092024-07-26T02:46:40.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10102024-07-26T02:46:40.750ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10112024-07-26T02:46:40.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
10122024-07-26T02:46:40.751ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10132024-07-26T02:46:40.751ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12959, tv_nsec: 715698666 } })
10142024-07-26T02:46:40.751ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
10152024-07-26T02:46:40.751ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
10162024-07-26T02:46:40.751ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 12.614205ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
10172024-07-26T02:46:40.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10182024-07-26T02:46:40.752ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
10192024-07-26T02:46:40.752ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 13.409863ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10202024-07-26T02:46:40.752ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
10212024-07-26T02:46:40.752ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10222024-07-26T02:46:40.753ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10232024-07-26T02:46:40.753ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
10242024-07-26T02:46:40.753ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 14.359685ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
10252024-07-26T02:46:40.753ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10262024-07-26T02:46:40.753ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10272024-07-26T02:46:40.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10282024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10292024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10302024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
10312024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
10322024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
10332024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10342024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10352024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10362024-07-26T02:46:40.754ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10372024-07-26T02:46:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10382024-07-26T02:46:40.755ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10392024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10402024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10412024-07-26T02:46:40.756ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
10422024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10432024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10442024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10452024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10462024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10472024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10482024-07-26T02:46:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10492024-07-26T02:46:40.757ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10502024-07-26T02:46:40.757ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10512024-07-26T02:46:40.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10522024-07-26T02:46:40.759ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10532024-07-26T02:46:40.759ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
10542024-07-26T02:46:40.760ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
10552024-07-26T02:46:40.760ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
10562024-07-26T02:46:40.760ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
10572024-07-26T02:46:40.760ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10582024-07-26T02:46:40.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10592024-07-26T02:46:40.761ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10602024-07-26T02:46:40.761ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10612024-07-26T02:46:40.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10622024-07-26T02:46:40.761ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10632024-07-26T02:46:40.761ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10642024-07-26T02:46:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10652024-07-26T02:46:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10662024-07-26T02:46:40.762ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10672024-07-26T02:46:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10682024-07-26T02:46:40.762ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10692024-07-26T02:46:40.762ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
10702024-07-26T02:46:40.762ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10712024-07-26T02:46:40.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10722024-07-26T02:46:40.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10732024-07-26T02:46:40.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10742024-07-26T02:46:40.764ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
10752024-07-26T02:46:40.764ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 0 version = 2
10762024-07-26T02:46:40.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10772024-07-26T02:46:40.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10782024-07-26T02:46:40.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10792024-07-26T02:46:40.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10802024-07-26T02:46:40.766ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
10812024-07-26T02:46:40.766ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 27.045611ms iteration = 18 status = {"physical_disks_added":0}
10822024-07-26T02:46:40.766ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10832024-07-26T02:46:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10842024-07-26T02:46:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
10852024-07-26T02:46:40.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10862024-07-26T02:46:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10872024-07-26T02:46:40.767ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10882024-07-26T02:46:40.768ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
10892024-07-26T02:46:40.768ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
10902024-07-26T02:46:40.768ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10912024-07-26T02:46:40.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10922024-07-26T02:46:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
10932024-07-26T02:46:40.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10942024-07-26T02:46:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10952024-07-26T02:46:40.769ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10962024-07-26T02:46:40.770ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
10972024-07-26T02:46:40.770ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10982024-07-26T02:46:40.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10992024-07-26T02:46:40.771ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11002024-07-26T02:46:40.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11012024-07-26T02:46:40.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11022024-07-26T02:46:40.772ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11032024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 1 version = 2
11042024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
11052024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-07-26 02:38:55.456872 UTC dns_group = external generation = 2
11062024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 34.366153ms iteration = 9 status = {"generation":2}
11072024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11082024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11092024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11102024-07-26T02:46:40.773ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11112024-07-26T02:46:40.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11122024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11132024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11142024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
11152024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
11162024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-07-26 02:38:54.868472 UTC dns_group = internal generation = 1
11172024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 36.498792ms iteration = 9 status = {"generation":1}
11182024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11192024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11202024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11212024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
11222024-07-26T02:46:40.775ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 36.871701ms iteration = 49 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
11232024-07-26T02:46:40.777ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11242024-07-26T02:46:40.777ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0
11252024-07-26T02:46:40.777ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:189 found = 0 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0
11262024-07-26T02:46:40.777ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 38.634972ms iteration = 9 status = {"error":null,"error_count":0,"found":0,"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0}
11272024-07-26T02:46:40.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
11282024-07-26T02:46:40.779ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11292024-07-26T02:46:40.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11302024-07-26T02:46:40.780ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 41.870808ms iteration = 17 status = 0
11312024-07-26T02:46:40.798ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-07-26T02:38:40.312512Z, time_modified: 2024-07-26T02:38:40.312512Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }) }
11322024-07-26T02:46:40.798ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 60.105573ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]}},"default_endpoint":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
11332024-07-26T02:46:40.934ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
11342024-07-26T02:46:40.993ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11352024-07-26T02:46:40.993ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "98643c16-f4dc-4229-88c8-e43c971572ff", "content-length": "372", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
11362024-07-26T02:46:40.993ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11372024-07-26T02:46:40.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11382024-07-26T02:46:40.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11392024-07-26T02:46:40.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11402024-07-26T02:46:40.996ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
11412024-07-26T02:46:40.996ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11422024-07-26T02:46:40.997ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "17665674-af21-4567-8dbd-884706e868d7", "content-length": "26", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
11432024-07-26T02:46:40.997ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11442024-07-26T02:46:40.998ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11452024-07-26T02:46:40.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11462024-07-26T02:46:40.998ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
11472024-07-26T02:46:41.000ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11482024-07-26T02:46:41.000ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "c3be1661-5529-4003-b897-34a87d6959a4", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
11492024-07-26T02:46:41.001ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 262.849188ms iteration = 17 status = {}
11502024-07-26T02:46:41.047ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
11512024-07-26T02:46:41.054ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
11522024-07-26T02:46:41.054ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11532024-07-26T02:46:41.054ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0350981f-a5fc-486d-ab62-1713431b9839", "content-length": "26", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
11542024-07-26T02:46:41.054ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11552024-07-26T02:46:41.056ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11562024-07-26T02:46:41.103ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
11572024-07-26T02:46:41.104ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11582024-07-26T02:46:41.104ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c43ed129-042d-4ff6-9615-e990d18223c0", "content-length": "26", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
11592024-07-26T02:46:41.104ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11602024-07-26T02:46:41.106ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11612024-07-26T02:46:41.110ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
11622024-07-26T02:46:41.111ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ef98feab-6be7-452c-b6c9-c6aa7fa273ca", "content-length": "691", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
11632024-07-26T02:46:41.111ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
11642024-07-26T02:46:41.111ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
11652024-07-26T02:46:41.111ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
11662024-07-26T02:46:41.114ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 375.517919ms iteration = 17 status = {}
11672024-07-26T02:46:41.118ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
11682024-07-26T02:46:41.119ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "7a51acb7-ab5b-4f0f-91c8-7c5bc60fc3ac", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
11692024-07-26T02:46:41.119ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
11702024-07-26T02:46:41.120ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "ca8aaea9-5efb-4a44-9c7f-b9ab3b2ed716", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
11712024-07-26T02:46:41.120ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
11722024-07-26T02:46:41.121ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
11732024-07-26T02:46:41.121ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 383.121086ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
11742024-07-26T02:46:41.191ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11752024-07-26T02:46:41.191ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ac79339d-71f6-4370-a480-7df33fbc1a80", "content-length": "2", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
11762024-07-26T02:46:41.193ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11772024-07-26T02:46:41.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11782024-07-26T02:46:41.195ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 456.647113ms iteration = 17 status = null
11792024-07-26T02:46:41.196ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
11802024-07-26T02:46:41.218ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 480.05792ms iteration = 17 status = {}
11812024-07-26T02:46:41.264ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
11822024-07-26T02:46:41.264ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "101f4d18-23bb-4fb7-ac53-9ac5f33368c7", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11832024-07-26T02:46:41.265ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11842024-07-26T02:46:41.266ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f9db3438-ab2e-4679-9cdb-73413ffacd68", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11852024-07-26T02:46:41.266ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11862024-07-26T02:46:41.268ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "210725c1-5069-4d05-b436-86615da612b8", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11872024-07-26T02:46:41.268ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11882024-07-26T02:46:41.270ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
11892024-07-26T02:46:41.270ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 532.117615ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
11902024-07-26T02:46:41.286ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11912024-07-26T02:46:41.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11922024-07-26T02:46:41.288ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
11932024-07-26T02:46:41.288ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11942024-07-26T02:46:41.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11952024-07-26T02:46:41.296ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
11962024-07-26T02:46:41.296ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d72d1a24-134a-4c33-95b2-5a9e4d11a6d5", "content-length": "121", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} })
11972024-07-26T02:46:41.296ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
11982024-07-26T02:46:41.296ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
11992024-07-26T02:46:41.296ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
12002024-07-26T02:46:41.296ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
12012024-07-26T02:46:41.296ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
12022024-07-26T02:46:41.296ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
12032024-07-26T02:46:41.296ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
12042024-07-26T02:46:41.296ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12052024-07-26T02:46:41.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12062024-07-26T02:46:41.298ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
12072024-07-26T02:46:41.298ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2b12ecd5-82d7-42ec-8cf2-5e0141e0950f", "content-length": "2", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} })
12082024-07-26T02:46:41.298ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
12092024-07-26T02:46:41.299ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "06a34bf8-dcff-4f14-963e-27c949d16ea0", "content-length": "52", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} })
12102024-07-26T02:46:41.299ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
12112024-07-26T02:46:41.299ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
12122024-07-26T02:46:41.347ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "10045563-9448-43ac-9d6d-2b33e3e5fbb6", "date": "Fri, 26 Jul 2024 02:46:40 GMT"} })
12132024-07-26T02:46:41.347ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
12142024-07-26T02:46:41.347ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
12152024-07-26T02:46:41.350ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a17ff0d7-48bf-4692-836b-5eea988c81ec", "content-length": "494", "date": "Fri, 26 Jul 2024 02:46:41 GMT"} })
12162024-07-26T02:46:41.350ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12172024-07-26T02:46:41.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12182024-07-26T02:46:41.353ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12192024-07-26T02:46:41.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12202024-07-26T02:46:41.356ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12212024-07-26T02:46:41.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12222024-07-26T02:46:41.357ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12232024-07-26T02:46:41.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12242024-07-26T02:46:41.364ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 625.772442ms iteration = 17 status = {}
12252024-07-26T02:46:41.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 486 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = b40b4318-882c-40ad-90b3-1df02af79792 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
12262024-07-26T02:46:50.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
12272024-07-26T02:46:50.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
12282024-07-26T02:46:50.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
12292024-07-26T02:46:50.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12302024-07-26T02:46:50.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12312024-07-26T02:46:50.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12322024-07-26T02:46:50.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12332024-07-26T02:46:50.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12342024-07-26T02:46:50.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12352024-07-26T02:46:50.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12362024-07-26T02:46:50.742ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
12372024-07-26T02:46:50.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.210395ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12382024-07-26T02:46:50.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12392024-07-26T02:46:50.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12402024-07-26T02:46:50.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12412024-07-26T02:46:50.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12422024-07-26T02:46:50.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12432024-07-26T02:46:50.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12442024-07-26T02:46:50.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
12452024-07-26T02:46:50.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12462024-07-26T02:46:50.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12472024-07-26T02:46:50.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
12482024-07-26T02:46:50.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.950516ms iteration = 50 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
12492024-07-26T02:46:51.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 442 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 780bf663-eeaf-4f64-8e95-e8d3e1d295a3 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
12502024-07-26T02:47:00.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
12512024-07-26T02:47:00.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
12522024-07-26T02:47:00.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12532024-07-26T02:47:00.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
12542024-07-26T02:47:00.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12552024-07-26T02:47:00.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12562024-07-26T02:47:00.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12572024-07-26T02:47:00.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12582024-07-26T02:47:00.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12592024-07-26T02:47:00.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12602024-07-26T02:47:00.742ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
12612024-07-26T02:47:00.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.41475ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12622024-07-26T02:47:00.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12632024-07-26T02:47:00.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12642024-07-26T02:47:00.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12652024-07-26T02:47:00.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12662024-07-26T02:47:00.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12672024-07-26T02:47:00.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12682024-07-26T02:47:00.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
12692024-07-26T02:47:00.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12702024-07-26T02:47:00.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12712024-07-26T02:47:00.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
12722024-07-26T02:47:00.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.282983ms iteration = 51 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
12732024-07-26T02:47:01.498ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 364 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = bba387c3-d999-4dcc-b120-aaf7ef990fe0 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
12742024-07-26T02:47:10.738ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
12752024-07-26T02:47:10.738ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
12762024-07-26T02:47:10.738ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
12772024-07-26T02:47:10.738ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12782024-07-26T02:47:10.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
12792024-07-26T02:47:10.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12802024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
12812024-07-26T02:47:10.739ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
12822024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12832024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
12842024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
12852024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12862024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12872024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
12882024-07-26T02:47:10.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
12892024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
12902024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12912024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12922024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
12932024-07-26T02:47:10.739ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
12942024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
12952024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
12962024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12972024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12982024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
12992024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
13002024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13012024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13022024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13032024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13042024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13052024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13062024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13072024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13082024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13092024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13102024-07-26T02:47:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13112024-07-26T02:47:10.741ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12989, tv_nsec: 705818279 } })
13122024-07-26T02:47:10.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13132024-07-26T02:47:10.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13142024-07-26T02:47:10.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13152024-07-26T02:47:10.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13162024-07-26T02:47:10.741ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
13172024-07-26T02:47:10.742ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12989, tv_nsec: 706705960 } })
13182024-07-26T02:47:10.742ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13192024-07-26T02:47:10.742ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13202024-07-26T02:47:10.742ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13212024-07-26T02:47:10.742ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13222024-07-26T02:47:10.742ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
13232024-07-26T02:47:10.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
13242024-07-26T02:47:10.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.88326ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13252024-07-26T02:47:10.745ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
13262024-07-26T02:47:10.745ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.114952ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
13272024-07-26T02:47:10.745ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
13282024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13292024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13302024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13312024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13322024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13332024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13342024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
13352024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13362024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
13372024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13382024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13392024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
13402024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13412024-07-26T02:47:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13422024-07-26T02:47:10.747ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
13432024-07-26T02:47:10.747ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.644949ms iteration = 18 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
13442024-07-26T02:47:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13452024-07-26T02:47:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13462024-07-26T02:47:10.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13472024-07-26T02:47:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13482024-07-26T02:47:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13492024-07-26T02:47:10.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13502024-07-26T02:47:10.748ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13512024-07-26T02:47:10.748ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
13522024-07-26T02:47:10.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13532024-07-26T02:47:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13542024-07-26T02:47:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13552024-07-26T02:47:10.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13562024-07-26T02:47:10.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13572024-07-26T02:47:10.749ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13582024-07-26T02:47:10.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13592024-07-26T02:47:10.749ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13602024-07-26T02:47:10.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13612024-07-26T02:47:10.750ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 12989, tv_nsec: 714693876 } })
13622024-07-26T02:47:10.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13632024-07-26T02:47:10.750ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
13642024-07-26T02:47:10.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13652024-07-26T02:47:10.752ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13662024-07-26T02:47:10.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13672024-07-26T02:47:10.753ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
13682024-07-26T02:47:10.753ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 13.812973ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
13692024-07-26T02:47:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
13702024-07-26T02:47:10.753ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13712024-07-26T02:47:10.753ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13722024-07-26T02:47:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13732024-07-26T02:47:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13742024-07-26T02:47:10.756ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
13752024-07-26T02:47:10.756ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 16.68325ms iteration = 19 status = {"physical_disks_added":0}
13762024-07-26T02:47:10.760ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
13772024-07-26T02:47:10.760ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.108803ms iteration = 52 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
13782024-07-26T02:47:10.762ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13792024-07-26T02:47:10.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13802024-07-26T02:47:10.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13812024-07-26T02:47:10.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13822024-07-26T02:47:10.765ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13832024-07-26T02:47:10.767ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
13842024-07-26T02:47:10.767ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13852024-07-26T02:47:10.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13862024-07-26T02:47:10.769ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 29.608871ms iteration = 18 status = 0
13872024-07-26T02:47:10.786ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13882024-07-26T02:47:10.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13892024-07-26T02:47:10.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13902024-07-26T02:47:10.789ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13912024-07-26T02:47:10.789ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13922024-07-26T02:47:10.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13932024-07-26T02:47:10.790ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13942024-07-26T02:47:10.792ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13952024-07-26T02:47:10.792ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13962024-07-26T02:47:10.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13972024-07-26T02:47:10.881ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
13982024-07-26T02:47:10.881ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13992024-07-26T02:47:10.882ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "bed3c7a1-7aa5-400e-adaa-fd528a2cf5e1", "content-length": "26", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14002024-07-26T02:47:10.882ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14012024-07-26T02:47:10.883ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14022024-07-26T02:47:10.995ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
14032024-07-26T02:47:10.995ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14042024-07-26T02:47:10.995ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "14ee0ed1-d7be-44ce-bc78-bde6f31143ee", "content-length": "26", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14052024-07-26T02:47:10.995ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14062024-07-26T02:47:10.996ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14072024-07-26T02:47:11.008ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14082024-07-26T02:47:11.008ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3c5a58ae-588d-49cb-93fd-a9ea598c36fd", "content-length": "372", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14092024-07-26T02:47:11.008ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14102024-07-26T02:47:11.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14112024-07-26T02:47:11.010ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14122024-07-26T02:47:11.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14132024-07-26T02:47:11.013ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14142024-07-26T02:47:11.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14152024-07-26T02:47:11.015ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
14162024-07-26T02:47:11.015ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14172024-07-26T02:47:11.015ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "86a5c63b-47ae-473d-89ce-f4380c55182b", "content-length": "26", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14182024-07-26T02:47:11.015ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14192024-07-26T02:47:11.016ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14202024-07-26T02:47:11.016ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
14212024-07-26T02:47:11.017ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f233395a-3b1c-4791-9fd2-b671e8bdfc81", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14222024-07-26T02:47:11.018ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 279.19133ms iteration = 18 status = {}
14232024-07-26T02:47:11.115ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
14242024-07-26T02:47:11.115ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "48133ca5-2576-4cfb-b8bf-1c03f14d05d5", "content-length": "691", "date": "Fri, 26 Jul 2024 02:47:11 GMT"} })
14252024-07-26T02:47:11.115ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
14262024-07-26T02:47:11.115ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
14272024-07-26T02:47:11.115ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
14282024-07-26T02:47:11.117ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 378.102728ms iteration = 18 status = {}
14292024-07-26T02:47:11.121ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
14302024-07-26T02:47:11.121ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "72c2951a-747a-4e83-b650-cebd6289f758", "content-length": "2", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14312024-07-26T02:47:11.122ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14322024-07-26T02:47:11.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14332024-07-26T02:47:11.124ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 385.999073ms iteration = 18 status = null
14342024-07-26T02:47:11.171ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 433.519663ms iteration = 18 status = {}
14352024-07-26T02:47:11.239ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14362024-07-26T02:47:11.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14372024-07-26T02:47:11.241ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
14382024-07-26T02:47:11.241ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14392024-07-26T02:47:11.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14402024-07-26T02:47:11.248ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
14412024-07-26T02:47:11.248ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "40d5ffc9-f7c4-4a2f-8f35-a5e26672a2f6", "content-length": "121", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14422024-07-26T02:47:11.248ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
14432024-07-26T02:47:11.248ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
14442024-07-26T02:47:11.248ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
14452024-07-26T02:47:11.248ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
14462024-07-26T02:47:11.248ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
14472024-07-26T02:47:11.248ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
14482024-07-26T02:47:11.248ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
14492024-07-26T02:47:11.248ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14502024-07-26T02:47:11.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14512024-07-26T02:47:11.250ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
14522024-07-26T02:47:11.250ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2d86d910-cdfc-434d-acba-c67484d76fcf", "content-length": "2", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14532024-07-26T02:47:11.250ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
14542024-07-26T02:47:11.251ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0cba9b7d-a946-457b-8066-34f90cd8b8e8", "content-length": "52", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14552024-07-26T02:47:11.251ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
14562024-07-26T02:47:11.251ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
14572024-07-26T02:47:11.295ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6411fc2f-2723-4c5f-8fac-f6c1d0117177", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14582024-07-26T02:47:11.295ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
14592024-07-26T02:47:11.295ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14602024-07-26T02:47:11.295ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c3619c4d-5d13-4591-972f-f30821bd49cd", "content-length": "494", "date": "Fri, 26 Jul 2024 02:47:10 GMT"} })
14612024-07-26T02:47:11.295ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14622024-07-26T02:47:11.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14632024-07-26T02:47:11.298ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14642024-07-26T02:47:11.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14652024-07-26T02:47:11.300ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14662024-07-26T02:47:11.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14672024-07-26T02:47:11.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14682024-07-26T02:47:11.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14692024-07-26T02:47:11.309ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 569.774044ms iteration = 18 status = {}
14702024-07-26T02:47:11.497ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 576 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 8eef45c5-002d-4766-8e2a-65ab4d71fde0 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
14712024-07-26T02:47:11.501ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
14722024-07-26T02:47:11.501ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
14732024-07-26T02:47:11.532ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
14742024-07-26T02:47:11.532ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:39660
14752024-07-26T02:47:11.532ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:39660 req_id = 583721ac-f123-4f00-a15a-aa50d86ac381 roles = RoleSet { roles: {} } uri = /metrics/producers
14762024-07-26T02:47:11.532ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:39660 req_id = 583721ac-f123-4f00-a15a-aa50d86ac381 resource = Database result = Ok(()) uri = /metrics/producers
14772024-07-26T02:47:11.564ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:338 id = 8752c2d7-36d8-49ed-860d-ec196a32b72c
14782024-07-26T02:47:11.564ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:39660 req_id = 583721ac-f123-4f00-a15a-aa50d86ac381 roles = RoleSet { roles: {} } uri = /metrics/producers
14792024-07-26T02:47:11.564ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:39660 req_id = 583721ac-f123-4f00-a15a-aa50d86ac381 resource = Database result = Ok(()) uri = /metrics/producers
14802024-07-26T02:47:11.570ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = 8752c2d7-36d8-49ed-860d-ec196a32b72c uri = http://[fd00:1122:3344:101::d]:12223/producers
14812024-07-26T02:47:11.571ZDEBGnexus: client response oximeter-collector = 8752c2d7-36d8-49ed-860d-ec196a32b72c result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "3fd6347b-770a-41dc-b21d-3826316ae404", "date": "Fri, 26 Jul 2024 02:47:11 GMT"} })
14822024-07-26T02:47:11.571ZINFOnexus: assigned collector to new producer collector_id = 8752c2d7-36d8-49ed-860d-ec196a32b72c file = nexus/src/app/oximeter.rs:125 producer_id = 9d80bc64-2ec6-4d4e-96d7-5c012d075087
14832024-07-26T02:47:11.575ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43052 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:39660 req_id = 583721ac-f123-4f00-a15a-aa50d86ac381 response_code = 201 uri = /metrics/producers
14842024-07-26T02:47:11.575ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 201, headers: {"content-type": "application/json", "x-request-id": "583721ac-f123-4f00-a15a-aa50d86ac381", "content-length": "41", "date": "Fri, 26 Jul 2024 02:47:11 GMT"} })
14852024-07-26T02:47:11.576ZDEBGnexus (producer-registration-task): registered with nexus successfully
14862024-07-26T02:47:11.576ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
14872024-07-26T02:47:20.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
14882024-07-26T02:47:20.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
14892024-07-26T02:47:20.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
14902024-07-26T02:47:20.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14912024-07-26T02:47:20.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14922024-07-26T02:47:20.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14932024-07-26T02:47:20.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14942024-07-26T02:47:20.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14952024-07-26T02:47:20.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14962024-07-26T02:47:20.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14972024-07-26T02:47:20.742ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
14982024-07-26T02:47:20.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.096866ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14992024-07-26T02:47:20.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15002024-07-26T02:47:20.743ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15012024-07-26T02:47:20.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15022024-07-26T02:47:20.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15032024-07-26T02:47:20.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15042024-07-26T02:47:20.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15052024-07-26T02:47:20.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
15062024-07-26T02:47:20.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15072024-07-26T02:47:20.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15082024-07-26T02:47:20.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
15092024-07-26T02:47:20.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.386866ms iteration = 53 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
15102024-07-26T02:47:21.573ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 627 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 828e50ca-f460-4121-9f42-4b21ee361ca4 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
15112024-07-26T02:47:30.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
15122024-07-26T02:47:30.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
15132024-07-26T02:47:30.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
15142024-07-26T02:47:30.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15152024-07-26T02:47:30.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15162024-07-26T02:47:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15172024-07-26T02:47:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15182024-07-26T02:47:30.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15192024-07-26T02:47:30.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15202024-07-26T02:47:30.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15212024-07-26T02:47:30.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
15222024-07-26T02:47:30.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.531066ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15232024-07-26T02:47:30.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15242024-07-26T02:47:30.743ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15252024-07-26T02:47:30.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15262024-07-26T02:47:30.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15272024-07-26T02:47:30.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15282024-07-26T02:47:30.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15292024-07-26T02:47:30.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
15302024-07-26T02:47:30.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15312024-07-26T02:47:30.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15322024-07-26T02:47:30.755ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
15332024-07-26T02:47:30.755ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.198112ms iteration = 54 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
15342024-07-26T02:47:31.572ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 592 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = e0136b9a-c3e4-4c3b-a6ee-dd5e274f0550 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
15352024-07-26T02:47:40.737ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
15362024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
15372024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15382024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
15392024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
15402024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
15412024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
15422024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
15432024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15442024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
15452024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
15462024-07-26T02:47:40.738ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2024-07-26 02:37:40.738092258 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
15472024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
15482024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
15492024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
15502024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15512024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
15522024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
15532024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15542024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15552024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
15562024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15572024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
15582024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
15592024-07-26T02:47:40.738ZINFOnexus (BackgroundTasks): lookup region port task started background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:56
15602024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15612024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
15622024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15632024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
15642024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
15652024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
15662024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15672024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15682024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
15692024-07-26T02:47:40.738ZINFOnexus (BackgroundTasks): Abandoned VMM reaper running background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:76
15702024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
15712024-07-26T02:47:40.738ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15722024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
15732024-07-26T02:47:40.739ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
15742024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15752024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
15762024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
15772024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15782024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
15792024-07-26T02:47:40.739ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:69 reason = blueprint disabled target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a
15802024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
15812024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.81\u{b5}s iteration = 11 status = {"error":"blueprint disabled","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a"}
15822024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
15832024-07-26T02:47:40.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
15842024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15852024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15862024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
15872024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15882024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
15892024-07-26T02:47:40.739ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
15902024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
15912024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
15922024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
15932024-07-26T02:47:40.739ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
15942024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
15952024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
15962024-07-26T02:47:40.739ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("25724e7e-f0f3-413d-ab41-b4a33973050b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4e64b0cb-e3e2-4959-adf4-d5cd70f25b40.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 13019, tv_nsec: 703790205 } })
15972024-07-26T02:47:40.739ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53979846-b429-4bae-9968-d96ff860d07e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 13019, tv_nsec: 703820476 } })
15982024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
15992024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16002024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16012024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16022024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
16032024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16042024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
16052024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16062024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16072024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16082024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16092024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16102024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16112024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16122024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16132024-07-26T02:47:40.739ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16142024-07-26T02:47:40.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16152024-07-26T02:47:40.740ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16162024-07-26T02:47:40.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16172024-07-26T02:47:40.740ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16182024-07-26T02:47:40.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16192024-07-26T02:47:40.740ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16202024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16212024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
16222024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 3.221675ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
16232024-07-26T02:47:40.741ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2024-07-26 02:37:40.738092258 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
16242024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.114473ms iteration = 10 status = {"expiration":"2024-07-26T02:37:40.738092258Z","pruned":[]}
16252024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16262024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16272024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
16282024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16292024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 3.452621ms iteration = 10 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
16302024-07-26T02:47:40.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16312024-07-26T02:47:40.742ZINFOnexus (BackgroundTasks): lookup region port task done background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:150
16322024-07-26T02:47:40.742ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 3.669106ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
16332024-07-26T02:47:40.742ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16342024-07-26T02:47:40.742ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16352024-07-26T02:47:40.742ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362024-07-26T02:47:40.743ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
16372024-07-26T02:47:40.743ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.419593ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
16382024-07-26T02:47:40.743ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13019, tv_nsec: 708170337 } })
16392024-07-26T02:47:40.744ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16402024-07-26T02:47:40.744ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16412024-07-26T02:47:40.744ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13019, tv_nsec: 709558050 } })
16422024-07-26T02:47:40.745ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
16432024-07-26T02:47:40.745ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
16442024-07-26T02:47:40.745ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16452024-07-26T02:47:40.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16462024-07-26T02:47:40.745ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
16472024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
16482024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16492024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16502024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
16512024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
16522024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16532024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16542024-07-26T02:47:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16552024-07-26T02:47:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
16562024-07-26T02:47:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16572024-07-26T02:47:40.747ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16582024-07-26T02:47:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16592024-07-26T02:47:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
16602024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16612024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16622024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
16632024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16642024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16652024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16662024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16672024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16682024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16692024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16702024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
16712024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
16722024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
16732024-07-26T02:47:40.748ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16742024-07-26T02:47:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16752024-07-26T02:47:40.750ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16762024-07-26T02:47:40.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
16772024-07-26T02:47:40.750ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
16782024-07-26T02:47:40.751ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16792024-07-26T02:47:40.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16802024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
16812024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
16822024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
16832024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16842024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16852024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16862024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16872024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16882024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16892024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
16902024-07-26T02:47:40.752ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16912024-07-26T02:47:40.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16922024-07-26T02:47:40.753ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
16932024-07-26T02:47:40.753ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16942024-07-26T02:47:40.753ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16952024-07-26T02:47:40.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16962024-07-26T02:47:40.754ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16972024-07-26T02:47:40.754ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16982024-07-26T02:47:40.754ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16992024-07-26T02:47:40.754ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
17002024-07-26T02:47:40.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17012024-07-26T02:47:40.754ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17022024-07-26T02:47:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
17032024-07-26T02:47:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17042024-07-26T02:47:40.756ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
17052024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 17.437507ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
17062024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17072024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17082024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17092024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17102024-07-26T02:47:40.756ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
17112024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 17.594391ms iteration = 19 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
17122024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17132024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17142024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
17152024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17162024-07-26T02:47:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17172024-07-26T02:47:40.757ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17182024-07-26T02:47:40.757ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
17192024-07-26T02:47:40.757ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17202024-07-26T02:47:40.758ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17212024-07-26T02:47:40.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17222024-07-26T02:47:40.758ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17232024-07-26T02:47:40.758ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
17242024-07-26T02:47:40.758ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 0 version = 2
17252024-07-26T02:47:40.758ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17262024-07-26T02:47:40.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17272024-07-26T02:47:40.759ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17282024-07-26T02:47:40.760ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
17292024-07-26T02:47:40.760ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17302024-07-26T02:47:40.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17312024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
17322024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17332024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
17342024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17352024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
17362024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17372024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17382024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
17392024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
17402024-07-26T02:47:40.762ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17412024-07-26T02:47:40.762ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
17422024-07-26T02:47:40.763ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 24.026581ms iteration = 20 status = {"physical_disks_added":0}
17432024-07-26T02:47:40.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17442024-07-26T02:47:40.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17452024-07-26T02:47:40.764ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17462024-07-26T02:47:40.765ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
17472024-07-26T02:47:40.765ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17482024-07-26T02:47:40.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
17492024-07-26T02:47:40.766ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17502024-07-26T02:47:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17512024-07-26T02:47:40.766ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17522024-07-26T02:47:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17532024-07-26T02:47:40.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17542024-07-26T02:47:40.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17552024-07-26T02:47:40.767ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17562024-07-26T02:47:40.768ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
17572024-07-26T02:47:40.768ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17582024-07-26T02:47:40.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17592024-07-26T02:47:40.768ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 1 version = 2
17602024-07-26T02:47:40.768ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
17612024-07-26T02:47:40.768ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-07-26 02:38:55.456872 UTC dns_group = external generation = 2
17622024-07-26T02:47:40.768ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 30.869221ms iteration = 10 status = {"generation":2}
17632024-07-26T02:47:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
17642024-07-26T02:47:40.769ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17652024-07-26T02:47:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17662024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
17672024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 31.295191ms iteration = 55 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
17682024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
17692024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17702024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17712024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17722024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
17732024-07-26T02:47:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17742024-07-26T02:47:40.771ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17752024-07-26T02:47:40.771ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
17762024-07-26T02:47:40.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17772024-07-26T02:47:40.771ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
17782024-07-26T02:47:40.771ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-07-26 02:38:54.868472 UTC dns_group = internal generation = 1
17792024-07-26T02:47:40.771ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 33.322188ms iteration = 10 status = {"generation":1}
17802024-07-26T02:47:40.771ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17812024-07-26T02:47:40.772ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17822024-07-26T02:47:40.772ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17832024-07-26T02:47:40.772ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
17842024-07-26T02:47:40.772ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.862851ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17852024-07-26T02:47:40.772ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17862024-07-26T02:47:40.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17872024-07-26T02:47:40.774ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17882024-07-26T02:47:40.775ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17892024-07-26T02:47:40.775ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17902024-07-26T02:47:40.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17912024-07-26T02:47:40.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
17922024-07-26T02:47:40.775ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17932024-07-26T02:47:40.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
17942024-07-26T02:47:40.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
17952024-07-26T02:47:40.776ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17962024-07-26T02:47:40.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17972024-07-26T02:47:40.776ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 38.002457ms iteration = 5 status = {"nsuccess":5}
17982024-07-26T02:47:40.778ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13019, tv_nsec: 742809736 } })
17992024-07-26T02:47:40.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
18002024-07-26T02:47:40.778ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18012024-07-26T02:47:40.778ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
18022024-07-26T02:47:40.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18032024-07-26T02:47:40.780ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 41.795466ms iteration = 19 status = 0
18042024-07-26T02:47:40.782ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0
18052024-07-26T02:47:40.782ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:189 found = 0 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0
18062024-07-26T02:47:40.782ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 44.058529ms iteration = 10 status = {"error":null,"error_count":0,"found":0,"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0}
18072024-07-26T02:47:40.783ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-07-26T02:38:40.312512Z, time_modified: 2024-07-26T02:38:40.312512Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }) }
18082024-07-26T02:47:40.783ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 45.800199ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]}},"default_endpoint":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
18092024-07-26T02:47:40.784ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
18102024-07-26T02:47:40.785ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
18112024-07-26T02:47:40.785ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 46.220759ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
18122024-07-26T02:47:40.914ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
18132024-07-26T02:47:40.920ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
18142024-07-26T02:47:40.920ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18152024-07-26T02:47:40.921ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "80e8b0cd-4248-4e5a-aeeb-b933c01398b9", "content-length": "26", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18162024-07-26T02:47:40.921ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18172024-07-26T02:47:40.922ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18182024-07-26T02:47:40.928ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
18192024-07-26T02:47:40.928ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18202024-07-26T02:47:40.928ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "cc23d7d9-3155-4869-a0af-61a602ef828c", "content-length": "26", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18212024-07-26T02:47:40.928ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18222024-07-26T02:47:40.929ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18232024-07-26T02:47:40.934ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
18242024-07-26T02:47:40.998ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
18252024-07-26T02:47:40.998ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18262024-07-26T02:47:40.999ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c49c88dc-6d1e-477d-8730-97422242e355", "content-length": "26", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18272024-07-26T02:47:40.999ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18282024-07-26T02:47:41.000ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
18292024-07-26T02:47:41.108ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 370.707746ms iteration = 19 status = {}
18302024-07-26T02:47:41.131ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
18312024-07-26T02:47:41.132ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9df4194e-222f-4e87-9dfd-3d3a07755e00", "content-length": "372", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18322024-07-26T02:47:41.132ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18332024-07-26T02:47:41.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18342024-07-26T02:47:41.134ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18352024-07-26T02:47:41.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18362024-07-26T02:47:41.137ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18372024-07-26T02:47:41.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18382024-07-26T02:47:41.137ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
18392024-07-26T02:47:41.137ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "dc35e20f-e5b8-4f0f-b173-23a7d845fc36", "content-length": "691", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18402024-07-26T02:47:41.137ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
18412024-07-26T02:47:41.138ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
18422024-07-26T02:47:41.138ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
18432024-07-26T02:47:41.140ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 401.236249ms iteration = 19 status = {}
18442024-07-26T02:47:41.140ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
18452024-07-26T02:47:41.140ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "7ff18b6e-2013-4c4f-996e-7baa6f59a907", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18462024-07-26T02:47:41.142ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 403.530332ms iteration = 19 status = {}
18472024-07-26T02:47:41.148ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
18482024-07-26T02:47:41.149ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "044c347f-1a85-4ac4-ab6a-f5fb59a95db7", "date": "Fri, 26 Jul 2024 02:47:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18492024-07-26T02:47:41.149ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18502024-07-26T02:47:41.150ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d004dfb3-8040-417b-9c92-798c60437576", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18512024-07-26T02:47:41.150ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18522024-07-26T02:47:41.152ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
18532024-07-26T02:47:41.152ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 414.350665ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
18542024-07-26T02:47:41.206ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
18552024-07-26T02:47:41.230ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
18562024-07-26T02:47:41.231ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c4cd230b-19d9-4faa-b517-7c66a5ffbe55", "content-length": "2", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18572024-07-26T02:47:41.233ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18582024-07-26T02:47:41.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
18592024-07-26T02:47:41.235ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 497.045917ms iteration = 19 status = null
18602024-07-26T02:47:41.285ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
18612024-07-26T02:47:41.285ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "e448c215-2c5c-4c33-a5f2-7f2bc5135c25", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18622024-07-26T02:47:41.285ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18632024-07-26T02:47:41.287ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "c513ce5f-4a0e-45cd-8d3e-4bf835e8613d", "date": "Fri, 26 Jul 2024 02:47:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18642024-07-26T02:47:41.287ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18652024-07-26T02:47:41.289ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4722930f-29ee-49b9-83e0-2eeb05add2a5", "date": "Fri, 26 Jul 2024 02:47:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18662024-07-26T02:47:41.289ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18672024-07-26T02:47:41.290ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
18682024-07-26T02:47:41.290ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 552.948291ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
18692024-07-26T02:47:41.303ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18702024-07-26T02:47:41.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18712024-07-26T02:47:41.305ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
18722024-07-26T02:47:41.305ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18732024-07-26T02:47:41.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18742024-07-26T02:47:41.311ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
18752024-07-26T02:47:41.311ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b9817c9f-8ddc-46da-814a-9fa1687bd678", "content-length": "121", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18762024-07-26T02:47:41.311ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
18772024-07-26T02:47:41.311ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
18782024-07-26T02:47:41.311ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
18792024-07-26T02:47:41.311ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
18802024-07-26T02:47:41.311ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
18812024-07-26T02:47:41.311ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
18822024-07-26T02:47:41.311ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
18832024-07-26T02:47:41.311ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18842024-07-26T02:47:41.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18852024-07-26T02:47:41.313ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
18862024-07-26T02:47:41.314ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "07bdac42-b94a-4397-ad60-f27ae1ba9eaf", "content-length": "2", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18872024-07-26T02:47:41.314ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
18882024-07-26T02:47:41.314ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "bf33d666-3bd4-4230-84e5-9a619bee98b7", "content-length": "52", "date": "Fri, 26 Jul 2024 02:47:40 GMT"} })
18892024-07-26T02:47:41.314ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
18902024-07-26T02:47:41.314ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
18912024-07-26T02:47:41.356ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5b3547f9-96de-4b9e-bf52-b0dc8eb990cd", "date": "Fri, 26 Jul 2024 02:47:41 GMT"} })
18922024-07-26T02:47:41.356ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
18932024-07-26T02:47:41.356ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
18942024-07-26T02:47:41.359ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8dfb6b3d-63cd-47b9-9e70-0f8a8c8d06c4", "content-length": "494", "date": "Fri, 26 Jul 2024 02:47:41 GMT"} })
18952024-07-26T02:47:41.359ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18962024-07-26T02:47:41.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18972024-07-26T02:47:41.361ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18982024-07-26T02:47:41.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18992024-07-26T02:47:41.364ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19002024-07-26T02:47:41.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19012024-07-26T02:47:41.365ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19022024-07-26T02:47:41.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19032024-07-26T02:47:41.371ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 632.951289ms iteration = 19 status = {}
19042024-07-26T02:47:41.573ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 539 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = c82d4231-362d-4a41-9eed-5e5e04c2b21b response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
19052024-07-26T02:47:50.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
19062024-07-26T02:47:50.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
19072024-07-26T02:47:50.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
19082024-07-26T02:47:50.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19092024-07-26T02:47:50.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19102024-07-26T02:47:50.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19112024-07-26T02:47:50.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19122024-07-26T02:47:50.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19132024-07-26T02:47:50.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19142024-07-26T02:47:50.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19152024-07-26T02:47:50.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
19162024-07-26T02:47:50.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.206109ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19172024-07-26T02:47:50.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19182024-07-26T02:47:50.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19192024-07-26T02:47:50.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19202024-07-26T02:47:50.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19212024-07-26T02:47:50.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19222024-07-26T02:47:50.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19232024-07-26T02:47:50.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
19242024-07-26T02:47:50.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19252024-07-26T02:47:50.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19262024-07-26T02:47:50.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
19272024-07-26T02:47:50.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.868194ms iteration = 56 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
19282024-07-26T02:47:51.572ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 411 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 13f24e55-3824-4ed1-a620-6926d8a23f5c response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
19292024-07-26T02:48:00.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
19302024-07-26T02:48:00.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
19312024-07-26T02:48:00.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
19322024-07-26T02:48:00.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19332024-07-26T02:48:00.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19342024-07-26T02:48:00.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19352024-07-26T02:48:00.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19362024-07-26T02:48:00.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19372024-07-26T02:48:00.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19382024-07-26T02:48:00.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19392024-07-26T02:48:00.742ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
19402024-07-26T02:48:00.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.423923ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19412024-07-26T02:48:00.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19422024-07-26T02:48:00.743ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19432024-07-26T02:48:00.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19442024-07-26T02:48:00.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19452024-07-26T02:48:00.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19462024-07-26T02:48:00.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19472024-07-26T02:48:00.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
19482024-07-26T02:48:00.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19492024-07-26T02:48:00.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19502024-07-26T02:48:00.755ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
19512024-07-26T02:48:00.755ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.494158ms iteration = 57 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
19522024-07-26T02:48:01.572ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 386 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 260aebbf-9bde-4d93-b9e9-e18882f1a0e0 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
19532024-07-26T02:48:10.738ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
19542024-07-26T02:48:10.738ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19552024-07-26T02:48:10.738ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
19562024-07-26T02:48:10.738ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
19572024-07-26T02:48:10.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19582024-07-26T02:48:10.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
19592024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
19602024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
19612024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19622024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
19632024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19642024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19652024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
19662024-07-26T02:48:10.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
19672024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19682024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
19692024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
19702024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19712024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
19722024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19732024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19742024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
19752024-07-26T02:48:10.739ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
19762024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
19772024-07-26T02:48:10.739ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
19782024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19792024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
19802024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19812024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19822024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19832024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19842024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19852024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19862024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19872024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19882024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19892024-07-26T02:48:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19902024-07-26T02:48:10.740ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19912024-07-26T02:48:10.740ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13049, tv_nsec: 705624712 } })
19922024-07-26T02:48:10.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19932024-07-26T02:48:10.741ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19942024-07-26T02:48:10.741ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
19952024-07-26T02:48:10.742ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19962024-07-26T02:48:10.742ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13049, tv_nsec: 706907613 } })
19972024-07-26T02:48:10.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19982024-07-26T02:48:10.742ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19992024-07-26T02:48:10.742ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20002024-07-26T02:48:10.743ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
20012024-07-26T02:48:10.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
20022024-07-26T02:48:10.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.424323ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20032024-07-26T02:48:10.743ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
20042024-07-26T02:48:10.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
20052024-07-26T02:48:10.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
20062024-07-26T02:48:10.744ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20072024-07-26T02:48:10.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20082024-07-26T02:48:10.744ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
20092024-07-26T02:48:10.744ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.375325ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
20102024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20112024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20122024-07-26T02:48:10.745ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
20132024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 5.854237ms iteration = 20 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
20142024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20152024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
20162024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20172024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20182024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20192024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
20202024-07-26T02:48:10.745ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
20212024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
20222024-07-26T02:48:10.746ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13049, tv_nsec: 711121331 } })
20232024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20242024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20252024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20262024-07-26T02:48:10.746ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
20272024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20282024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20292024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20302024-07-26T02:48:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20312024-07-26T02:48:10.747ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
20322024-07-26T02:48:10.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
20332024-07-26T02:48:10.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20342024-07-26T02:48:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20352024-07-26T02:48:10.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20362024-07-26T02:48:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20372024-07-26T02:48:10.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20382024-07-26T02:48:10.748ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20392024-07-26T02:48:10.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20402024-07-26T02:48:10.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20412024-07-26T02:48:10.750ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20422024-07-26T02:48:10.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20432024-07-26T02:48:10.751ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
20442024-07-26T02:48:10.751ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20452024-07-26T02:48:10.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20462024-07-26T02:48:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
20472024-07-26T02:48:10.753ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20482024-07-26T02:48:10.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20492024-07-26T02:48:10.753ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20502024-07-26T02:48:10.753ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
20512024-07-26T02:48:10.754ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 14.850456ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
20522024-07-26T02:48:10.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20532024-07-26T02:48:10.760ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
20542024-07-26T02:48:10.760ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.347899ms iteration = 58 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
20552024-07-26T02:48:10.761ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20562024-07-26T02:48:10.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20572024-07-26T02:48:10.761ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20582024-07-26T02:48:10.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20592024-07-26T02:48:10.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20602024-07-26T02:48:10.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20612024-07-26T02:48:10.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20622024-07-26T02:48:10.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20632024-07-26T02:48:10.765ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20642024-07-26T02:48:10.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20652024-07-26T02:48:10.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20662024-07-26T02:48:10.766ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20672024-07-26T02:48:10.767ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
20682024-07-26T02:48:10.767ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20692024-07-26T02:48:10.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20702024-07-26T02:48:10.768ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20712024-07-26T02:48:10.768ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20722024-07-26T02:48:10.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20732024-07-26T02:48:10.768ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 29.506459ms iteration = 20 status = 0
20742024-07-26T02:48:10.779ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
20752024-07-26T02:48:10.779ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 39.928582ms iteration = 21 status = {"physical_disks_added":0}
20762024-07-26T02:48:10.879ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
20772024-07-26T02:48:10.879ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20782024-07-26T02:48:10.879ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7f1eb9ae-5421-4d55-b54f-c5828b5fdd7a", "content-length": "26", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
20792024-07-26T02:48:10.879ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20802024-07-26T02:48:10.881ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
20812024-07-26T02:48:10.908ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20822024-07-26T02:48:10.909ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ea410b42-7da2-41e4-a89d-2f4150e3bef4", "content-length": "372", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
20832024-07-26T02:48:10.909ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20842024-07-26T02:48:10.909ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20852024-07-26T02:48:10.911ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20862024-07-26T02:48:10.911ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20872024-07-26T02:48:10.913ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20882024-07-26T02:48:10.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20892024-07-26T02:48:10.916ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20902024-07-26T02:48:10.916ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a3341963-9ef9-4eb8-9512-0b4f3efe48a0", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
20912024-07-26T02:48:10.917ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 178.037017ms iteration = 20 status = {}
20922024-07-26T02:48:10.923ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
20932024-07-26T02:48:10.924ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "714d2095-a4a9-442e-8197-285336c1a813", "content-length": "691", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
20942024-07-26T02:48:10.924ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
20952024-07-26T02:48:10.924ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
20962024-07-26T02:48:10.924ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
20972024-07-26T02:48:10.925ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 186.360782ms iteration = 20 status = {}
20982024-07-26T02:48:10.982ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
20992024-07-26T02:48:10.982ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21002024-07-26T02:48:10.982ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1d72e45c-56e8-4caa-8e81-8f0f52f869b9", "content-length": "26", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
21012024-07-26T02:48:10.982ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21022024-07-26T02:48:10.983ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
21032024-07-26T02:48:10.983ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
21042024-07-26T02:48:10.983ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21052024-07-26T02:48:10.984ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "478d0929-8c9e-4e36-ba6e-f5214c344b37", "content-length": "26", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
21062024-07-26T02:48:10.984ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21072024-07-26T02:48:10.984ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4b264dc7-879a-4727-874f-a6fa20fb4767", "content-length": "2", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
21082024-07-26T02:48:10.986ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21092024-07-26T02:48:10.987ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
21102024-07-26T02:48:10.987ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21112024-07-26T02:48:10.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21122024-07-26T02:48:10.988ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 250.878168ms iteration = 20 status = null
21132024-07-26T02:48:11.048ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 310.819008ms iteration = 20 status = {}
21142024-07-26T02:48:11.106ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21152024-07-26T02:48:11.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21162024-07-26T02:48:11.108ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
21172024-07-26T02:48:11.108ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21182024-07-26T02:48:11.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21192024-07-26T02:48:11.115ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
21202024-07-26T02:48:11.115ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2980f362-a570-4356-9566-a4768ddafc10", "content-length": "121", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
21212024-07-26T02:48:11.115ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
21222024-07-26T02:48:11.115ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
21232024-07-26T02:48:11.116ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
21242024-07-26T02:48:11.116ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
21252024-07-26T02:48:11.116ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
21262024-07-26T02:48:11.116ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
21272024-07-26T02:48:11.116ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
21282024-07-26T02:48:11.116ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21292024-07-26T02:48:11.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21302024-07-26T02:48:11.117ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
21312024-07-26T02:48:11.117ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b32ade10-f6a1-41ac-aad5-f420706409a5", "content-length": "2", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
21322024-07-26T02:48:11.117ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
21332024-07-26T02:48:11.117ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "768bda14-d67c-4e76-a2df-75dbc6f42a89", "content-length": "52", "date": "Fri, 26 Jul 2024 02:48:10 GMT"} })
21342024-07-26T02:48:11.117ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
21352024-07-26T02:48:11.117ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
21362024-07-26T02:48:11.162ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "457d0343-ad04-4ef8-864e-6c49cff19376", "date": "Fri, 26 Jul 2024 02:48:11 GMT"} })
21372024-07-26T02:48:11.162ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
21382024-07-26T02:48:11.162ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
21392024-07-26T02:48:11.164ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f68e1cdd-6570-46ba-b180-186384f2f951", "content-length": "494", "date": "Fri, 26 Jul 2024 02:48:11 GMT"} })
21402024-07-26T02:48:11.164ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21412024-07-26T02:48:11.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21422024-07-26T02:48:11.167ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21432024-07-26T02:48:11.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21442024-07-26T02:48:11.170ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21452024-07-26T02:48:11.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21462024-07-26T02:48:11.172ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21472024-07-26T02:48:11.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21482024-07-26T02:48:11.178ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 439.268867ms iteration = 20 status = {}
21492024-07-26T02:48:11.572ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 941 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 5d382d65-f070-434b-a653-5cd99d117be6 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
21502024-07-26T02:48:20.739ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
21512024-07-26T02:48:20.739ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
21522024-07-26T02:48:20.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21532024-07-26T02:48:20.739ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
21542024-07-26T02:48:20.739ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21552024-07-26T02:48:20.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21562024-07-26T02:48:20.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21572024-07-26T02:48:20.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21582024-07-26T02:48:20.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21592024-07-26T02:48:20.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21602024-07-26T02:48:20.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
21612024-07-26T02:48:20.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.022844ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21622024-07-26T02:48:20.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21632024-07-26T02:48:20.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21642024-07-26T02:48:20.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21652024-07-26T02:48:20.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21662024-07-26T02:48:20.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21672024-07-26T02:48:20.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21682024-07-26T02:48:20.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
21692024-07-26T02:48:20.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21702024-07-26T02:48:20.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21712024-07-26T02:48:20.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
21722024-07-26T02:48:20.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.231233ms iteration = 59 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
21732024-07-26T02:48:21.572ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 415 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 33b3296f-d2d2-4ff9-992d-42ba2fd59199 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
21742024-07-26T02:48:30.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
21752024-07-26T02:48:30.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
21762024-07-26T02:48:30.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21772024-07-26T02:48:30.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
21782024-07-26T02:48:30.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21792024-07-26T02:48:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21802024-07-26T02:48:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21812024-07-26T02:48:30.740ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21822024-07-26T02:48:30.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21832024-07-26T02:48:30.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21842024-07-26T02:48:30.742ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
21852024-07-26T02:48:30.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.170357ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21862024-07-26T02:48:30.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21872024-07-26T02:48:30.743ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21882024-07-26T02:48:30.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21892024-07-26T02:48:30.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21902024-07-26T02:48:30.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21912024-07-26T02:48:30.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21922024-07-26T02:48:30.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
21932024-07-26T02:48:30.751ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21942024-07-26T02:48:30.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21952024-07-26T02:48:30.757ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
21962024-07-26T02:48:30.757ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.574804ms iteration = 60 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
21972024-07-26T02:48:31.572ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1056 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 2c5ce460-501d-4c30-80e2-85ee50e08d90 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
21982024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
21992024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
22002024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
22012024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22022024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
22032024-07-26T02:48:40.737ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2024-07-26 02:38:40.737193039 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
22042024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
22052024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
22062024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
22072024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
22082024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22092024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
22102024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
22112024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
22122024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22132024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
22142024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
22152024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22162024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
22172024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22182024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
22192024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
22202024-07-26T02:48:40.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22212024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22222024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22232024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
22242024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
22252024-07-26T02:48:40.738ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
22262024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22272024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
22282024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
22292024-07-26T02:48:40.738ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
22302024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
22312024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
22322024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22332024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
22342024-07-26T02:48:40.738ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53979846-b429-4bae-9968-d96ff860d07e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cf7587f8-33d7-4e89-bad7-ae8cdd556425.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 13079, tv_nsec: 703122854 } })
22352024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
22362024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22372024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22382024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
22392024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22402024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
22412024-07-26T02:48:40.738ZINFOnexus (BackgroundTasks): Abandoned VMM reaper running background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:76
22422024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
22432024-07-26T02:48:40.738ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("25724e7e-f0f3-413d-ab41-b4a33973050b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4e64b0cb-e3e2-4959-adf4-d5cd70f25b40.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("82379658-84e2-487b-bcd4-b30ffc1d9874.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 13079, tv_nsec: 703155435 } })
22442024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
22452024-07-26T02:48:40.738ZINFOnexus (BackgroundTasks): lookup region port task started background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:56
22462024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
22472024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
22482024-07-26T02:48:40.738ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
22492024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
22502024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
22512024-07-26T02:48:40.738ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:69 reason = blueprint disabled target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a
22522024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 18.75\u{b5}s iteration = 12 status = {"error":"blueprint disabled","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a"}
22532024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
22542024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
22552024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22562024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
22572024-07-26T02:48:40.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
22582024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22592024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22602024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
22612024-07-26T02:48:40.738ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
22622024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
22632024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
22642024-07-26T02:48:40.738ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
22652024-07-26T02:48:40.738ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
22662024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
22672024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
22682024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22692024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22702024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
22712024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22722024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22732024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22742024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22752024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
22762024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22772024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
22782024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
22792024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22802024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22812024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
22822024-07-26T02:48:40.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22832024-07-26T02:48:40.738ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
22842024-07-26T02:48:40.739ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
22852024-07-26T02:48:40.739ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 2.12095ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
22862024-07-26T02:48:40.739ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22872024-07-26T02:48:40.740ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22882024-07-26T02:48:40.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22892024-07-26T02:48:40.740ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22902024-07-26T02:48:40.740ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
22912024-07-26T02:48:40.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
22922024-07-26T02:48:40.740ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
22932024-07-26T02:48:40.740ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 3.604204ms iteration = 11 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
22942024-07-26T02:48:40.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22952024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22962024-07-26T02:48:40.742ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2024-07-26 02:38:40.737193039 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
22972024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.875204ms iteration = 11 status = {"expiration":"2024-07-26T02:38:40.737193039Z","pruned":[]}
22982024-07-26T02:48:40.742ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13079, tv_nsec: 707129668 } })
22992024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23002024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23012024-07-26T02:48:40.742ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13079, tv_nsec: 707311842 } })
23022024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
23032024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23042024-07-26T02:48:40.742ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
23052024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.156148ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
23062024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23072024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23082024-07-26T02:48:40.742ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
23092024-07-26T02:48:40.743ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
23102024-07-26T02:48:40.743ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
23112024-07-26T02:48:40.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
23122024-07-26T02:48:40.743ZINFOnexus (BackgroundTasks): lookup region port task done background_task = lookup_region_port file = nexus/src/app/background/tasks/lookup_region_port.rs:150
23132024-07-26T02:48:40.743ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 5.209551ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
23142024-07-26T02:48:40.743ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23152024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23162024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23172024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23182024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
23192024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
23202024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23212024-07-26T02:48:40.744ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
23222024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 6.630055ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23232024-07-26T02:48:40.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
23242024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0
23252024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23262024-07-26T02:48:40.745ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:189 found = 0 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0
23272024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 6.896451ms iteration = 11 status = {"error":null,"error_count":0,"found":0,"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0}
23282024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
23292024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23302024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23312024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23322024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23332024-07-26T02:48:40.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23342024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23352024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23362024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
23372024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23382024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23392024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
23402024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23412024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23422024-07-26T02:48:40.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23432024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
23442024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23452024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23462024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23472024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23482024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
23492024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
23502024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
23512024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23522024-07-26T02:48:40.747ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
23532024-07-26T02:48:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
23542024-07-26T02:48:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23552024-07-26T02:48:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23562024-07-26T02:48:40.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23572024-07-26T02:48:40.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23582024-07-26T02:48:40.748ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13079, tv_nsec: 713693741 } })
23592024-07-26T02:48:40.749ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
23602024-07-26T02:48:40.749ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23612024-07-26T02:48:40.749ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23622024-07-26T02:48:40.749ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23632024-07-26T02:48:40.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
23642024-07-26T02:48:40.749ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23652024-07-26T02:48:40.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23662024-07-26T02:48:40.750ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
23672024-07-26T02:48:40.750ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23682024-07-26T02:48:40.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23692024-07-26T02:48:40.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
23702024-07-26T02:48:40.751ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23712024-07-26T02:48:40.751ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23722024-07-26T02:48:40.751ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23732024-07-26T02:48:40.751ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23742024-07-26T02:48:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23752024-07-26T02:48:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23762024-07-26T02:48:40.752ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23772024-07-26T02:48:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
23782024-07-26T02:48:40.752ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
23792024-07-26T02:48:40.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
23802024-07-26T02:48:40.753ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
23812024-07-26T02:48:40.753ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 15.661116ms iteration = 21 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
23822024-07-26T02:48:40.755ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23832024-07-26T02:48:40.755ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
23842024-07-26T02:48:40.755ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
23852024-07-26T02:48:40.755ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 17.487909ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
23862024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
23872024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
23882024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
23892024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23902024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
23912024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
23922024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23932024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23942024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23952024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
23962024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23972024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(f2609def-40f6-47b0-be18-0d4aa3ec29b3, 0), (3823c580-4048-470b-970b-dfb4fa37202d, 0), (7a6badc2-9096-4e5f-9259-b96934740855, 0), (03c7bb30-7e71-45a1-8700-7379f76b5a3e, 0)] collection_id = f2609def-40f6-47b0-be18-0d4aa3ec29b3
23982024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23992024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
24002024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24012024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
24022024-07-26T02:48:40.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24032024-07-26T02:48:40.757ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24042024-07-26T02:48:40.757ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24052024-07-26T02:48:40.757ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24062024-07-26T02:48:40.757ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24072024-07-26T02:48:40.757ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
24082024-07-26T02:48:40.757ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 19.669239ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
24092024-07-26T02:48:40.758ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
24102024-07-26T02:48:40.758ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 0 version = 2
24112024-07-26T02:48:40.758ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24122024-07-26T02:48:40.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24132024-07-26T02:48:40.758ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24142024-07-26T02:48:40.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24152024-07-26T02:48:40.759ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24162024-07-26T02:48:40.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24172024-07-26T02:48:40.759ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24182024-07-26T02:48:40.760ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24192024-07-26T02:48:40.760ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
24202024-07-26T02:48:40.760ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 22.082855ms iteration = 22 status = {"physical_disks_added":0}
24212024-07-26T02:48:40.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24222024-07-26T02:48:40.760ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24232024-07-26T02:48:40.761ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24242024-07-26T02:48:40.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24252024-07-26T02:48:40.761ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24262024-07-26T02:48:40.762ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24272024-07-26T02:48:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
24282024-07-26T02:48:40.762ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24292024-07-26T02:48:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24302024-07-26T02:48:40.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24312024-07-26T02:48:40.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24322024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
24332024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24342024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24352024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24362024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
24372024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
24382024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24392024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24402024-07-26T02:48:40.764ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24412024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
24422024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
24432024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24442024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24452024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24462024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 8703d396-8e31-45b7-94b9-1a5aa1e7e591 dns_zone_name = oxide.test found_so_far = 1 version = 2
24472024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
24482024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-07-26 02:38:55.456872 UTC dns_group = external generation = 2
24492024-07-26T02:48:40.765ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 28.520916ms iteration = 11 status = {"generation":2}
24502024-07-26T02:48:40.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24512024-07-26T02:48:40.766ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24522024-07-26T02:48:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
24532024-07-26T02:48:40.766ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24542024-07-26T02:48:40.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24552024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
24562024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24572024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
24582024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24592024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24602024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24612024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
24622024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24632024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
24642024-07-26T02:48:40.769ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24652024-07-26T02:48:40.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24662024-07-26T02:48:40.770ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24672024-07-26T02:48:40.770ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 32.708154ms iteration = 21 status = 0
24682024-07-26T02:48:40.771ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24692024-07-26T02:48:40.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24702024-07-26T02:48:40.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24712024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
24722024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 33.856721ms iteration = 61 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
24732024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 2bd64804-7bee-4b66-b318-ffd3693239b2 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
24742024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
24752024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-07-26 02:38:54.868472 UTC dns_group = internal generation = 1
24762024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 35.180251ms iteration = 11 status = {"generation":1}
24772024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24782024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24792024-07-26T02:48:40.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24802024-07-26T02:48:40.773ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24812024-07-26T02:48:40.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24822024-07-26T02:48:40.775ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24832024-07-26T02:48:40.775ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24842024-07-26T02:48:40.775ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24852024-07-26T02:48:40.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24862024-07-26T02:48:40.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24872024-07-26T02:48:40.776ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24882024-07-26T02:48:40.777ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
24892024-07-26T02:48:40.777ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
24902024-07-26T02:48:40.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
24912024-07-26T02:48:40.777ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 39.036351ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
24922024-07-26T02:48:40.778ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24932024-07-26T02:48:40.779ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24942024-07-26T02:48:40.779ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24952024-07-26T02:48:40.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24962024-07-26T02:48:40.779ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-07-26T02:38:40.312512Z, time_modified: 2024-07-26T02:38:40.312512Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, db_silo: Silo { identity: SiloIdentity { id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-07-26T02:38:55.405921Z, time_modified: 2024-07-26T02:38:55.405921Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9" }] }) }
24972024-07-26T02:48:40.779ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 42.722347ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]}},"default_endpoint":{"silo_id":"2218f0bf-bef6-4785-b5b0-f8053a24df0b","tls_certs":["cee4bbdb858392d30ede3dfb833fc85654131355479d6db95df8518e1880c8a9"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
24982024-07-26T02:48:40.782ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
24992024-07-26T02:48:40.782ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25002024-07-26T02:48:40.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25012024-07-26T02:48:40.783ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25022024-07-26T02:48:40.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25032024-07-26T02:48:40.784ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25042024-07-26T02:48:40.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25052024-07-26T02:48:40.786ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25062024-07-26T02:48:40.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25072024-07-26T02:48:40.789ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25082024-07-26T02:48:40.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25092024-07-26T02:48:40.790ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25102024-07-26T02:48:40.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25112024-07-26T02:48:40.791ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25122024-07-26T02:48:40.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25132024-07-26T02:48:40.793ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25142024-07-26T02:48:40.796ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
25152024-07-26T02:48:40.796ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25162024-07-26T02:48:40.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25172024-07-26T02:48:40.797ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25182024-07-26T02:48:40.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25192024-07-26T02:48:40.799ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25202024-07-26T02:48:40.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25212024-07-26T02:48:40.800ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25222024-07-26T02:48:40.803ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
25232024-07-26T02:48:40.803ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25242024-07-26T02:48:40.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25252024-07-26T02:48:40.813ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25262024-07-26T02:48:40.813ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25272024-07-26T02:48:40.814ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25282024-07-26T02:48:40.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25292024-07-26T02:48:40.816ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25302024-07-26T02:48:40.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25312024-07-26T02:48:40.818ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25322024-07-26T02:48:40.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25332024-07-26T02:48:40.819ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25342024-07-26T02:48:40.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25352024-07-26T02:48:40.820ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25362024-07-26T02:48:40.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25372024-07-26T02:48:40.822ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25382024-07-26T02:48:40.822ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25392024-07-26T02:48:40.823ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25402024-07-26T02:48:40.827ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
25412024-07-26T02:48:40.827ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25422024-07-26T02:48:40.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25432024-07-26T02:48:40.828ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25442024-07-26T02:48:40.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25452024-07-26T02:48:40.829ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25462024-07-26T02:48:40.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25472024-07-26T02:48:40.830ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25482024-07-26T02:48:40.833ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
25492024-07-26T02:48:40.833ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25502024-07-26T02:48:40.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25512024-07-26T02:48:40.841ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = f2609def-40f6-47b0-be18-0d4aa3ec29b3 file = nexus/db-queries/src/db/datastore/inventory.rs:1299 ncabooses = 0 ncollections = 0 nerrors = 0 nnics = 0 nphysical_disks = 0 nrot_pages = 0 nrots = 0 nsled_agent_zones = 0 nsled_agents = 0 nsps = 0 nzones = 0 nzpools = 0
25522024-07-26T02:48:40.841ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
25532024-07-26T02:48:40.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
25542024-07-26T02:48:40.843ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(3823c580-4048-470b-970b-dfb4fa37202d, 0), (7a6badc2-9096-4e5f-9259-b96934740855, 0), (03c7bb30-7e71-45a1-8700-7379f76b5a3e, 0)]
25552024-07-26T02:48:40.844ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13079, tv_nsec: 808995487 } })
25562024-07-26T02:48:40.844ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
25572024-07-26T02:48:40.844ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25582024-07-26T02:48:40.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25592024-07-26T02:48:40.846ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25602024-07-26T02:48:40.848ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
25612024-07-26T02:48:40.848ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25622024-07-26T02:48:40.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25632024-07-26T02:48:40.849ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
25642024-07-26T02:48:40.849ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [VpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, VpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
25652024-07-26T02:48:40.849ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
25662024-07-26T02:48:40.881ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: ea0e0fc4-fc93-4de6-b687-98567d824e83, time_created: 2024-07-26T02:38:52.798018Z, time_modified: 2024-07-26T02:38:52.798018Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::1:0, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)) }]
25672024-07-26T02:48:40.881ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
25682024-07-26T02:48:40.881ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25692024-07-26T02:48:40.881ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25702024-07-26T02:48:40.883ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
25712024-07-26T02:48:40.883ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
25722024-07-26T02:48:40.885ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25732024-07-26T02:48:40.887ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
25742024-07-26T02:48:40.976ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
25752024-07-26T02:48:40.976ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25762024-07-26T02:48:40.976ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2096d756-ed8f-4c84-a79a-efbd28a81ead", "content-length": "26", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
25772024-07-26T02:48:40.976ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25782024-07-26T02:48:40.978ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
25792024-07-26T02:48:40.997ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
25802024-07-26T02:48:40.997ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25812024-07-26T02:48:40.998ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d6799b4b-6e9a-49af-8db0-c33e9f713d26", "content-length": "26", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
25822024-07-26T02:48:40.998ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25832024-07-26T02:48:41.000ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
25842024-07-26T02:48:41.017ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
25852024-07-26T02:48:41.017ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d60ad195-430a-473a-a34a-22f4b50d2f76", "content-length": "691", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
25862024-07-26T02:48:41.018ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
25872024-07-26T02:48:41.018ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
25882024-07-26T02:48:41.018ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
25892024-07-26T02:48:41.019ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 281.260998ms iteration = 21 status = {}
25902024-07-26T02:48:41.092ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
25912024-07-26T02:48:41.103ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
25922024-07-26T02:48:41.103ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
25932024-07-26T02:48:41.103ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
25942024-07-26T02:48:41.104ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/ignition", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d2da7773-2094-425c-a001-35926ebdc189", "content-length": "476", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
25952024-07-26T02:48:41.104ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
25962024-07-26T02:48:41.104ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "67181848-a3a8-4762-8597-baccaf95bd22", "content-length": "734", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
25972024-07-26T02:48:41.104ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
25982024-07-26T02:48:41.104ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/sp/caboose", query: Some("firmware_slot=0"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "152777ef-358e-4439-8a24-89e41ae7188a", "content-length": "86", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
25992024-07-26T02:48:41.104ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
26002024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/sp/caboose", query: Some("firmware_slot=1"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9cfa88cf-ea60-4e94-b5ed-43630b8846e2", "content-length": "86", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26012024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
26022024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/rot/caboose", query: Some("firmware_slot=0"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "cbe8a33b-abe6-469d-983e-32d566c50469", "content-length": "80", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26032024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
26042024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/rot/caboose", query: Some("firmware_slot=1"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "61aded67-2b10-45b0-b223-f2f77724627a", "content-length": "80", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26052024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
26062024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/stage0/caboose", query: Some("firmware_slot=0"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a94f5756-f696-444b-bb60-4c6b389320b6", "content-length": "88", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26072024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
26082024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/stage0/caboose", query: Some("firmware_slot=1"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "931db391-9c74-42e5-80ff-e6761ab99a63", "content-length": "88", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26092024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
26102024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/rot/cmpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8332c044-52e6-4af5-bac7-b26185302017", "content-length": "702", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26112024-07-26T02:48:41.106ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
26122024-07-26T02:48:41.107ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/rot/cfpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a348d73f-2480-41db-9006-3eb29802e1a5", "content-length": "727", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26132024-07-26T02:48:41.107ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
26142024-07-26T02:48:41.107ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/rot/cfpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9dbfcab3-e1fb-461c-89a0-dd70ae055734", "content-length": "729", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26152024-07-26T02:48:41.107ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
26162024-07-26T02:48:41.108ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/switch/0/component/rot/cfpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "bb305e75-82d4-4bf3-9b17-cef5370cd4c3", "content-length": "728", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26172024-07-26T02:48:41.108ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
26182024-07-26T02:48:41.108ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3dc816c5-ceaf-4eaf-aa9c-d53b23e82349", "content-length": "722", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26192024-07-26T02:48:41.108ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
26202024-07-26T02:48:41.108ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/sp/caboose", query: Some("firmware_slot=0"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "16ab4f02-7782-460c-8fd0-3d3f88bfc7f2", "content-length": "84", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26212024-07-26T02:48:41.108ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
26222024-07-26T02:48:41.109ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/sp/caboose", query: Some("firmware_slot=1"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c11873-71ed-45a7-898d-337372bf8c42", "content-length": "84", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26232024-07-26T02:48:41.109ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
26242024-07-26T02:48:41.109ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/rot/caboose", query: Some("firmware_slot=0"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8fdb47-157e-449f-8d3e-44030f996d8a", "content-length": "82", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26252024-07-26T02:48:41.109ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
26262024-07-26T02:48:41.109ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/rot/caboose", query: Some("firmware_slot=1"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4cff6478-3805-4485-8fb6-06ed3b99ff08", "content-length": "82", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26272024-07-26T02:48:41.109ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
26282024-07-26T02:48:41.110ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/stage0/caboose", query: Some("firmware_slot=0"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd6a66b-79e5-424e-bd92-d72dfe87379a", "content-length": "91", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26292024-07-26T02:48:41.110ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
26302024-07-26T02:48:41.110ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/stage0/caboose", query: Some("firmware_slot=1"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ffc8f38a-d61c-497d-8877-5b0f17ee17a9", "content-length": "91", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26312024-07-26T02:48:41.110ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
26322024-07-26T02:48:41.111ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/rot/cmpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b3e067d0-f2b9-4696-84e8-1556ade6cc90", "content-length": "702", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26332024-07-26T02:48:41.111ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
26342024-07-26T02:48:41.111ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/rot/cfpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ae669cd3-127c-44e0-8c3c-9c0c4da2376f", "content-length": "727", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26352024-07-26T02:48:41.111ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
26362024-07-26T02:48:41.111ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/rot/cfpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e25a48be-eb54-45c3-b721-ac9ca37585ec", "content-length": "729", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
26372024-07-26T02:48:41.111ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
26382024-07-26T02:48:41.112ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/sp/sled/0/component/rot/cfpa", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d42c3218-8744-4d64-b996-25e3c0b84df4", "content-length": "728", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26392024-07-26T02:48:41.112ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
26402024-07-26T02:48:41.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
26412024-07-26T02:48:41.114ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26422024-07-26T02:48:41.116ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
26432024-07-26T02:48:41.116ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
26442024-07-26T02:48:41.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
26452024-07-26T02:48:41.117ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26462024-07-26T02:48:41.119ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
26472024-07-26T02:48:41.119ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
26482024-07-26T02:48:41.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
26492024-07-26T02:48:41.164ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
26502024-07-26T02:48:41.164ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26512024-07-26T02:48:41.164ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7301ee8a-c813-4d9d-9861-7eef1ef9eebc", "content-length": "26", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26522024-07-26T02:48:41.164ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26532024-07-26T02:48:41.166ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26542024-07-26T02:48:41.167ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
26552024-07-26T02:48:41.313ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
26562024-07-26T02:48:41.314ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e607c487-de6b-459b-9852-845282fddf8b", "content-length": "372", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
26572024-07-26T02:48:41.314ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26582024-07-26T02:48:41.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26592024-07-26T02:48:41.316ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26602024-07-26T02:48:41.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26612024-07-26T02:48:41.318ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26622024-07-26T02:48:41.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26632024-07-26T02:48:41.321ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
26642024-07-26T02:48:41.321ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "965388c8-7a95-4256-a243-e35452a9ec0c", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
26652024-07-26T02:48:41.323ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 584.98705ms iteration = 21 status = {}
26662024-07-26T02:48:41.360ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
26672024-07-26T02:48:41.360ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
26682024-07-26T02:48:41.375ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 638.759536ms iteration = 21 status = {}
26692024-07-26T02:48:41.422ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
26702024-07-26T02:48:41.423ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ef72eda7-d6f0-4284-82d7-f57fc658c92d", "content-length": "2", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} })
26712024-07-26T02:48:41.425ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26722024-07-26T02:48:41.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26732024-07-26T02:48:41.426ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26742024-07-26T02:48:41.427ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 689.920181ms iteration = 21 status = null
26752024-07-26T02:48:41.428ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = service_firewall_rule_propagation result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", query: None, fragment: None }, status: 204, headers: {"x-request-id": "31c3fdfc-dd89-4588-b651-0491e2d08f3c", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
26762024-07-26T02:48:41.430ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:494
26772024-07-26T02:48:41.430ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 692.51077ms
26782024-07-26T02:48:41.430ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 692.51878ms iteration = 3 status = {}
26792024-07-26T02:48:41.458ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
26802024-07-26T02:48:41.459ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "002caf0c-d508-4c9e-96c7-18aa31b2214c", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26812024-07-26T02:48:41.459ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26822024-07-26T02:48:41.461ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "8780b7af-3e78-48c4-84ef-916a1a17e72b", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26832024-07-26T02:48:41.461ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26842024-07-26T02:48:41.462ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
26852024-07-26T02:48:41.463ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 725.628864ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
26862024-07-26T02:48:41.485ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
26872024-07-26T02:48:41.514ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/inventory", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e497729a-9fee-410b-89a1-e8cf4f0acac5", "content-length": "2360", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
26882024-07-26T02:48:41.526ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/omicron-zones
26892024-07-26T02:48:41.527ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/omicron-zones", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0ac71197-034a-4708-b068-9164c3bfb37c", "content-length": "10314", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
26902024-07-26T02:48:41.540ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
26912024-07-26T02:48:41.540ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
26922024-07-26T02:48:41.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
26932024-07-26T02:48:41.541ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26942024-07-26T02:48:41.543ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
26952024-07-26T02:48:41.546ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
26962024-07-26T02:48:41.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
26972024-07-26T02:48:41.572ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 375 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 5b16ce9c-3f4e-43e5-9fc3-b647135a7897 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
26982024-07-26T02:48:41.604ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
26992024-07-26T02:48:41.605ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "0864dd14-b469-487d-9c7b-b8488421d8fc", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27002024-07-26T02:48:41.605ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27012024-07-26T02:48:41.606ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "c6418652-17ab-43d1-81cb-aa7540ed8693", "date": "Fri, 26 Jul 2024 02:48:40 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27022024-07-26T02:48:41.606ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27032024-07-26T02:48:41.608ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a55d015c-36a4-4090-a163-1f2dac7b0508", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27042024-07-26T02:48:41.608ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27052024-07-26T02:48:41.610ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
27062024-07-26T02:48:41.610ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 873.123448ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
27072024-07-26T02:48:41.624ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27082024-07-26T02:48:41.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27092024-07-26T02:48:41.626ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
27102024-07-26T02:48:41.626ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27112024-07-26T02:48:41.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27122024-07-26T02:48:41.634ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
27132024-07-26T02:48:41.635ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a69e32b3-ac54-44bb-bd3f-2c092f157d36", "content-length": "121", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
27142024-07-26T02:48:41.635ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
27152024-07-26T02:48:41.635ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
27162024-07-26T02:48:41.635ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
27172024-07-26T02:48:41.635ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
27182024-07-26T02:48:41.635ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
27192024-07-26T02:48:41.635ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
27202024-07-26T02:48:41.635ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
27212024-07-26T02:48:41.635ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27222024-07-26T02:48:41.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27232024-07-26T02:48:41.637ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
27242024-07-26T02:48:41.637ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "44e26560-30c9-4d46-9cbd-392211f7e7dd", "content-length": "2", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
27252024-07-26T02:48:41.637ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
27262024-07-26T02:48:41.637ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "6b723c69-3832-4493-a974-54115afcdd9c", "content-length": "52", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
27272024-07-26T02:48:41.637ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
27282024-07-26T02:48:41.637ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
27292024-07-26T02:48:41.697ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "e8d915f9-4155-4d15-8361-3afeb4ea9be8", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
27302024-07-26T02:48:41.697ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
27312024-07-26T02:48:41.697ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
27322024-07-26T02:48:41.697ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7a350a38-6242-4eb3-b3d2-e1e77385bd30", "content-length": "494", "date": "Fri, 26 Jul 2024 02:48:41 GMT"} })
27332024-07-26T02:48:41.697ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27342024-07-26T02:48:41.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27352024-07-26T02:48:41.706ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27362024-07-26T02:48:41.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27372024-07-26T02:48:41.708ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27382024-07-26T02:48:41.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27392024-07-26T02:48:41.710ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27402024-07-26T02:48:41.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27412024-07-26T02:48:41.713ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 94ad8990-16ac-48ad-8d56-587571e14015 file = nexus/db-queries/src/db/datastore/inventory.rs:893
27422024-07-26T02:48:41.715ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 94ad8990-16ac-48ad-8d56-587571e14015 time_started = 2024-07-26 02:48:41.102572 UTC
27432024-07-26T02:48:41.715ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 976.920512ms iteration = 2 status = {"collection_id":"94ad8990-16ac-48ad-8d56-587571e14015","time_done":"2024-07-26 02:48:41.540195 UTC","time_started":"2024-07-26 02:48:41.102572 UTC"}
27442024-07-26T02:48:41.715ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
27452024-07-26T02:48:41.715ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27462024-07-26T02:48:41.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27472024-07-26T02:48:41.716ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27482024-07-26T02:48:41.718ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 980.446025ms iteration = 21 status = {}
27492024-07-26T02:48:41.719ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27502024-07-26T02:48:41.719ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27512024-07-26T02:48:41.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27522024-07-26T02:48:41.721ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
27532024-07-26T02:48:41.721ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27542024-07-26T02:48:41.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27552024-07-26T02:48:41.722ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27562024-07-26T02:48:41.725ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27572024-07-26T02:48:41.725ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27582024-07-26T02:48:41.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27592024-07-26T02:48:41.732ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
27602024-07-26T02:48:41.732ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 17.471208ms iteration = 23 status = {"physical_disks_added":0}
27612024-07-26T02:48:44.501ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:61391
27622024-07-26T02:48:44.501ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61391 req_id = 7b591cbf-6c1d-4d64-8fcb-2f461e606552 roles = RoleSet { roles: {} } uri = /metrics/collectors/8752c2d7-36d8-49ed-860d-ec196a32b72c/producers?sort_by=id_ascending
27632024-07-26T02:48:44.501ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61391 req_id = 7b591cbf-6c1d-4d64-8fcb-2f461e606552 resource = Database result = Ok(()) uri = /metrics/collectors/8752c2d7-36d8-49ed-860d-ec196a32b72c/producers?sort_by=id_ascending
27642024-07-26T02:48:44.509ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8404 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61391 req_id = 7b591cbf-6c1d-4d64-8fcb-2f461e606552 response_code = 200 uri = /metrics/collectors/8752c2d7-36d8-49ed-860d-ec196a32b72c/producers?sort_by=id_ascending
27652024-07-26T02:48:44.525ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61391 req_id = 13f5c022-c363-4961-9b9a-5ff3dd6c8ef4 roles = RoleSet { roles: {} } uri = /metrics/collectors/8752c2d7-36d8-49ed-860d-ec196a32b72c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlYTBlMGZjNC1mYzkzLTRkZTYtYjY4Ny05ODU2N2Q4MjRlODMifX0%3D
27662024-07-26T02:48:44.525ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61391 req_id = 13f5c022-c363-4961-9b9a-5ff3dd6c8ef4 resource = Database result = Ok(()) uri = /metrics/collectors/8752c2d7-36d8-49ed-860d-ec196a32b72c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlYTBlMGZjNC1mYzkzLTRkZTYtYjY4Ny05ODU2N2Q4MjRlODMifX0%3D
27672024-07-26T02:48:44.527ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 4825 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:61391 req_id = 13f5c022-c363-4961-9b9a-5ff3dd6c8ef4 response_code = 200 uri = /metrics/collectors/8752c2d7-36d8-49ed-860d-ec196a32b72c/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlYTBlMGZjNC1mYzkzLTRkZTYtYjY4Ny05ODU2N2Q4MjRlODMifX0%3D
27682024-07-26T02:48:50.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
27692024-07-26T02:48:50.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27702024-07-26T02:48:50.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
27712024-07-26T02:48:50.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
27722024-07-26T02:48:50.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27732024-07-26T02:48:50.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27742024-07-26T02:48:50.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27752024-07-26T02:48:50.740ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27762024-07-26T02:48:50.740ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27772024-07-26T02:48:50.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27782024-07-26T02:48:50.742ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
27792024-07-26T02:48:50.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.905041ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27802024-07-26T02:48:50.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27812024-07-26T02:48:50.743ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27822024-07-26T02:48:50.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27832024-07-26T02:48:50.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27842024-07-26T02:48:50.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27852024-07-26T02:48:50.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27862024-07-26T02:48:50.748ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
27872024-07-26T02:48:50.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27882024-07-26T02:48:50.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27892024-07-26T02:48:50.754ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
27902024-07-26T02:48:50.754ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.658459ms iteration = 62 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
27912024-07-26T02:48:54.529ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 420 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 5f2e0210-c331-4512-a063-29073cc1f867 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
27922024-07-26T02:49:00.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
27932024-07-26T02:49:00.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
27942024-07-26T02:49:00.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
27952024-07-26T02:49:00.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27962024-07-26T02:49:00.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27972024-07-26T02:49:00.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27982024-07-26T02:49:00.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27992024-07-26T02:49:00.740ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28002024-07-26T02:49:00.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28012024-07-26T02:49:00.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28022024-07-26T02:49:00.742ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
28032024-07-26T02:49:00.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.196618ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
28042024-07-26T02:49:00.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
28052024-07-26T02:49:00.743ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28062024-07-26T02:49:00.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28072024-07-26T02:49:00.745ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28082024-07-26T02:49:00.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28092024-07-26T02:49:00.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28102024-07-26T02:49:00.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
28112024-07-26T02:49:00.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28122024-07-26T02:49:00.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28132024-07-26T02:49:00.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
28142024-07-26T02:49:00.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.710314ms iteration = 63 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
28152024-07-26T02:49:04.529ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 404 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = 9d137cc6-0d3e-41ee-8f43-2dc919eeb020 response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087
28162024-07-26T02:49:10.737ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
28172024-07-26T02:49:10.737ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
28182024-07-26T02:49:10.737ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
28192024-07-26T02:49:10.737ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
28202024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
28212024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
28222024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
28232024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
28242024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
28252024-07-26T02:49:10.738ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:64
28262024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28272024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
28282024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28292024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
28302024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
28312024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28322024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28332024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
28342024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
28352024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
28362024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28372024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
28382024-07-26T02:49:10.738ZINFOnexus (BackgroundTasks): region replacement driver task started background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:231
28392024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
28402024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28412024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28422024-07-26T02:49:10.738ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
28432024-07-26T02:49:10.738ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:46
28442024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
28452024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28462024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28472024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28482024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28492024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
28502024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28512024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28522024-07-26T02:49:10.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28532024-07-26T02:49:10.740ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13109, tv_nsec: 705776178 } })
28542024-07-26T02:49:10.740ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28552024-07-26T02:49:10.740ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28562024-07-26T02:49:10.741ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1033
28572024-07-26T02:49:10.741ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28582024-07-26T02:49:10.741ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13109, tv_nsec: 706575347 } })
28592024-07-26T02:49:10.741ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28602024-07-26T02:49:10.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28612024-07-26T02:49:10.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28622024-07-26T02:49:10.742ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28632024-07-26T02:49:10.742ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1033
28642024-07-26T02:49:10.742ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28652024-07-26T02:49:10.742ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
28662024-07-26T02:49:10.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
28672024-07-26T02:49:10.743ZINFOnexus (BackgroundTasks): region replacement driver task done background_task = region_replacement_driver file = nexus/src/app/background/tasks/region_replacement_driver.rs:238
28682024-07-26T02:49:10.743ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.412453ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
28692024-07-26T02:49:10.744ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/tasks/phantom_disks.rs:93
28702024-07-26T02:49:10.744ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.117849ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
28712024-07-26T02:49:10.744ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/tasks/region_replacement.rs:185
28722024-07-26T02:49:10.744ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 5.358995ms iteration = 22 status = {"region_replacement_started_err":0,"region_replacement_started_ok":0}
28732024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
28742024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
28752024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28762024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28772024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
28782024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28792024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
28802024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28812024-07-26T02:49:10.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28822024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
28832024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28842024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28852024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
28862024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28872024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28882024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28892024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
28902024-07-26T02:49:10.746ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28912024-07-26T02:49:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28922024-07-26T02:49:10.747ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:100
28932024-07-26T02:49:10.747ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28942024-07-26T02:49:10.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28952024-07-26T02:49:10.747ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28962024-07-26T02:49:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28972024-07-26T02:49:10.748ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28982024-07-26T02:49:10.748ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ea0e0fc4-fc93-4de6-b687-98567d824e83.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 13109, tv_nsec: 713329815 } })
28992024-07-26T02:49:10.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29002024-07-26T02:49:10.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29012024-07-26T02:49:10.748ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
29022024-07-26T02:49:10.748ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29032024-07-26T02:49:10.748ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1033 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
29042024-07-26T02:49:10.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29052024-07-26T02:49:10.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29062024-07-26T02:49:10.750ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29072024-07-26T02:49:10.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29082024-07-26T02:49:10.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
29092024-07-26T02:49:10.750ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29102024-07-26T02:49:10.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29112024-07-26T02:49:10.751ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:458 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
29122024-07-26T02:49:10.751ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 12.368759ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0}
29132024-07-26T02:49:10.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 6c70f35f-f523-48d3-bd2c-a2b84bba151a, lookup_type: ById(6c70f35f-f523-48d3-bd2c-a2b84bba151a) } result = Ok(())
29142024-07-26T02:49:10.752ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29152024-07-26T02:49:10.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29162024-07-26T02:49:10.758ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29172024-07-26T02:49:10.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29182024-07-26T02:49:10.759ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a original_time_created = 2024-07-26 02:38:39.428427 UTC target_id = 6c70f35f-f523-48d3-bd2c-a2b84bba151a time_created = 2024-07-26 02:38:39.428427 UTC
29192024-07-26T02:49:10.759ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.239273ms iteration = 64 status = {"status":"target blueprint unchanged","target_id":"6c70f35f-f523-48d3-bd2c-a2b84bba151a","time_created":"2024-07-26T02:38:39.428427Z"}
29202024-07-26T02:49:10.759ZWARNnexus (BackgroundTasks): physical disk adoption task done background_task = physical_disk_adoption file = nexus/src/app/background/tasks/physical_disk_adoption.rs:174
29212024-07-26T02:49:10.759ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 20.195432ms iteration = 24 status = {"physical_disks_added":0}
29222024-07-26T02:49:10.759ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29232024-07-26T02:49:10.761ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
29242024-07-26T02:49:10.761ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29252024-07-26T02:49:10.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29262024-07-26T02:49:10.762ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29272024-07-26T02:49:10.764ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
29282024-07-26T02:49:10.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29292024-07-26T02:49:10.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29302024-07-26T02:49:10.766ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29312024-07-26T02:49:10.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29322024-07-26T02:49:10.768ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29332024-07-26T02:49:10.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29342024-07-26T02:49:10.769ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29352024-07-26T02:49:10.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ea0e0fc4-fc93-4de6-b687-98567d824e83, lookup_type: ById(ea0e0fc4-fc93-4de6-b687-98567d824e83) } result = Ok(())
29362024-07-26T02:49:10.772ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29372024-07-26T02:49:10.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29382024-07-26T02:49:10.773ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 34.848944ms iteration = 22 status = 0
29392024-07-26T02:49:10.869ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
29402024-07-26T02:49:10.869ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29412024-07-26T02:49:10.869ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2e0fa675-80cd-4022-92d3-f6df6ce41f46", "content-length": "26", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29422024-07-26T02:49:10.869ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29432024-07-26T02:49:10.871ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
29442024-07-26T02:49:10.908ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1041 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 zone_address = fd00:1122:3344:101::2
29452024-07-26T02:49:10.908ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29462024-07-26T02:49:10.908ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a66e1cca-5037-4b15-9b39-a8e254bf1cfc", "content-length": "26", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29472024-07-26T02:49:10.908ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1046 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29482024-07-26T02:49:10.908ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
29492024-07-26T02:49:10.909ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = v2p_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/v2p", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5fe6552c-d4b5-4341-8c7a-152020222849", "content-length": "691", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29502024-07-26T02:49:10.909ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
29512024-07-26T02:49:10.909ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
29522024-07-26T02:49:10.909ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
29532024-07-26T02:49:10.910ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
29542024-07-26T02:49:10.912ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 173.234876ms iteration = 22 status = {}
29552024-07-26T02:49:11.015ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1041 zone_address = fd00:1122:3344:101::2
29562024-07-26T02:49:11.015ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29572024-07-26T02:49:11.015ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "11811ec3-72f1-445b-80b6-6b551a2e143f", "content-length": "26", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29582024-07-26T02:49:11.015ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1046 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29592024-07-26T02:49:11.017ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1078 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
29602024-07-26T02:49:11.056ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29612024-07-26T02:49:11.090ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e8bcb0e4-2e75-4231-943e-c260f1af0bd4", "content-length": "372", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29622024-07-26T02:49:11.090ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
29632024-07-26T02:49:11.090ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29642024-07-26T02:49:11.090ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b2d397f3-378a-451d-8d96-95de7452b15c", "content-length": "2", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29652024-07-26T02:49:11.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29662024-07-26T02:49:11.092ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
29672024-07-26T02:49:11.092ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29682024-07-26T02:49:11.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
29692024-07-26T02:49:11.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29702024-07-26T02:49:11.093ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 356.214898ms iteration = 22 status = null
29712024-07-26T02:49:11.094ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29722024-07-26T02:49:11.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29732024-07-26T02:49:11.097ZDEBGnexus (BackgroundTasks): client request SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29742024-07-26T02:49:11.097ZDEBGnexus (BackgroundTasks): client response SledAgent = ea0e0fc4-fc93-4de6-b687-98567d824e83 background_task = vpc_route_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc-routes", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f6793160-9656-4bb9-a423-3b5d73f13dec", "date": "Fri, 26 Jul 2024 02:49:11 GMT"} })
29752024-07-26T02:49:11.098ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 359.923114ms iteration = 22 status = {}
29762024-07-26T02:49:11.100ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 362.998966ms iteration = 22 status = {}
29772024-07-26T02:49:11.132ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29782024-07-26T02:49:11.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29792024-07-26T02:49:11.134ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:146 port = SwitchPort { id: a6ce3cf4-7297-486b-bcab-94c6c1828624, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(b561b3c7-0991-4814-9efa-92011cb42066) } rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
29802024-07-26T02:49:11.134ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29812024-07-26T02:49:11.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29822024-07-26T02:49:11.141ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
29832024-07-26T02:49:11.141ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "bc0e7edc-a14f-4461-afe8-d0a9786b2244", "content-length": "121", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29842024-07-26T02:49:11.141ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1663 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
29852024-07-26T02:49:11.141ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1690 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = b561b3c7-0991-4814-9efa-92011cb42066
29862024-07-26T02:49:11.141ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
29872024-07-26T02:49:11.141ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
29882024-07-26T02:49:11.141ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
29892024-07-26T02:49:11.141ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 routes = {}
29902024-07-26T02:49:11.141ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7
29912024-07-26T02:49:11.141ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29922024-07-26T02:49:11.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29932024-07-26T02:49:11.143ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
29942024-07-26T02:49:11.144ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "49bd9c4c-0c89-469e-92d1-2d67cb6727cb", "content-length": "2", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29952024-07-26T02:49:11.144ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
29962024-07-26T02:49:11.144ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "fcea2590-8dd0-4daf-abc8-bfa686d77efa", "content-length": "52", "date": "Fri, 26 Jul 2024 02:49:10 GMT"} })
29972024-07-26T02:49:11.144ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], port: "qsfp0" }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch_location = Switch0
29982024-07-26T02:49:11.144ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
29992024-07-26T02:49:11.188ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "43e5d3f1-47ec-44e1-a5c0-aa6f29248d25", "date": "Fri, 26 Jul 2024 02:49:11 GMT"} })
30002024-07-26T02:49:11.188ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 switch = Switch0
30012024-07-26T02:49:11.188ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
30022024-07-26T02:49:11.192ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 6101b963-ee1b-430c-ad60-5a5922d708b7 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5266cb84-461c-469d-ab41-f9dd6fc185de", "content-length": "494", "date": "Fri, 26 Jul 2024 02:49:11 GMT"} })
30032024-07-26T02:49:11.192ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30042024-07-26T02:49:11.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30052024-07-26T02:49:11.194ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30062024-07-26T02:49:11.195ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30072024-07-26T02:49:11.197ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30082024-07-26T02:49:11.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30092024-07-26T02:49:11.199ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30102024-07-26T02:49:11.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30112024-07-26T02:49:11.205ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 466.649147ms iteration = 22 status = {}
30122024-07-26T02:49:14.528ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 398 local_addr = [fd00:1122:3344:101::c]:48098 method = GET remote_addr = [fd00:1122:3344:101::d]:35480 req_id = e398318a-79fc-4c08-bdb4-3c118614a7ba response_code = 200 uri = /9d80bc64-2ec6-4d4e-96d7-5c012d075087