Max level shown:
12024-02-16T03:00:03.285ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
22024-02-16T03:00:03.285ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
32024-02-16T03:00:03.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42024-02-16T03:00:03.285ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
52024-02-16T03:00:03.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
62024-02-16T03:00:03.285ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
72024-02-16T03:00:03.286ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
82024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 14.121\u{b5}s iteration = 9
92024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
102024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
112024-02-16T03:00:03.286ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
122024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
132024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
142024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
152024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
162024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
172024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
182024-02-16T03:00:03.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
192024-02-16T03:00:03.288ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
202024-02-16T03:00:03.288ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
212024-02-16T03:00:03.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
222024-02-16T03:00:03.289ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
232024-02-16T03:00:03.289ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
242024-02-16T03:00:03.289ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.155952ms iteration = 9
252024-02-16T03:00:03.289ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d48e4dd1-6015-4495-8f92-fbcd20240e10", "content-length": "2", "date": "Fri, 16 Feb 2024 03:00:03 GMT"} })
262024-02-16T03:00:03.289ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 4.076777ms iteration = 9
272024-02-16T03:00:03.289ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7c6f474b-326d-4e80-828d-730e37748a32", "content-length": "1", "date": "Fri, 16 Feb 2024 03:00:02 GMT"} })
282024-02-16T03:00:03.289ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
292024-02-16T03:00:03.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
302024-02-16T03:00:03.291ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 5.817346ms iteration = 9
312024-02-16T03:00:03.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
322024-02-16T03:00:03.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
332024-02-16T03:00:03.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
342024-02-16T03:00:03.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
352024-02-16T03:00:03.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
362024-02-16T03:00:03.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
372024-02-16T03:00:03.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
382024-02-16T03:00:03.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 7.533875ms iteration = 23
392024-02-16T03:00:03.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
402024-02-16T03:00:03.303ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
412024-02-16T03:00:03.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
422024-02-16T03:00:03.305ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
432024-02-16T03:00:03.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
442024-02-16T03:00:03.306ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
452024-02-16T03:00:03.308ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
462024-02-16T03:00:03.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
472024-02-16T03:00:03.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
482024-02-16T03:00:03.310ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 24.465003ms iteration = 9
492024-02-16T03:00:03.535ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 196 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = f278c4f0-f47d-4845-9611-052c20bde511 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
502024-02-16T03:00:13.285ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
512024-02-16T03:00:13.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
522024-02-16T03:00:13.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
532024-02-16T03:00:13.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
542024-02-16T03:00:13.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
552024-02-16T03:00:13.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
562024-02-16T03:00:13.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
572024-02-16T03:00:13.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.682434ms iteration = 24
582024-02-16T03:00:13.535ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 253 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 6e34038e-90bd-48a4-adfa-02e78ed1020b response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
592024-02-16T03:00:23.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
602024-02-16T03:00:23.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
612024-02-16T03:00:23.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
622024-02-16T03:00:23.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
632024-02-16T03:00:23.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
642024-02-16T03:00:23.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
652024-02-16T03:00:23.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
662024-02-16T03:00:23.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.757535ms iteration = 25
672024-02-16T03:00:23.535ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 169 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 90589685-109d-42ff-afe5-0c128098cc38 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
682024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
692024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
702024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
712024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
722024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
732024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
742024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
752024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
762024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
772024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
782024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
792024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
802024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
812024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
822024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
832024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
842024-02-16T03:00:33.285ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
852024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
862024-02-16T03:00:33.285ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
872024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
882024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10\u{b5}s iteration = 6
892024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
902024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
912024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
922024-02-16T03:00:33.285ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
932024-02-16T03:00:33.285ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
942024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
952024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.49\u{b5}s iteration = 10
962024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
972024-02-16T03:00:33.285ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
982024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
992024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1002024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
1012024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1022024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1032024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
1042024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1052024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1062024-02-16T03:00:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1072024-02-16T03:00:33.287ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1082024-02-16T03:00:33.287ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1092024-02-16T03:00:33.288ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1102024-02-16T03:00:33.288ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1112024-02-16T03:00:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1122024-02-16T03:00:33.288ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
1132024-02-16T03:00:33.288ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.058041ms iteration = 10
1142024-02-16T03:00:33.289ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
1152024-02-16T03:00:33.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1162024-02-16T03:00:33.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1172024-02-16T03:00:33.289ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "fe7695b4-ba4c-4eb6-bc5d-1dc24fb8539d", "content-length": "1", "date": "Fri, 16 Feb 2024 03:00:32 GMT"} })
1182024-02-16T03:00:33.289ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1192024-02-16T03:00:33.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1202024-02-16T03:00:33.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1212024-02-16T03:00:33.290ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1222024-02-16T03:00:33.290ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4e6a5dd0-ffa9-4a16-860d-d1502cf63104", "content-length": "2", "date": "Fri, 16 Feb 2024 03:00:33 GMT"} })
1232024-02-16T03:00:33.290ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 4.308581ms iteration = 10
1242024-02-16T03:00:33.290ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1252024-02-16T03:00:33.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1262024-02-16T03:00:33.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1272024-02-16T03:00:33.291ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1282024-02-16T03:00:33.291ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1292024-02-16T03:00:33.291ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 6.157361ms iteration = 10
1302024-02-16T03:00:33.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1312024-02-16T03:00:33.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
1322024-02-16T03:00:33.293ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1332024-02-16T03:00:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1342024-02-16T03:00:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Fleet result = Ok(())
1352024-02-16T03:00:33.294ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
1362024-02-16T03:00:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
1372024-02-16T03:00:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Fleet result = Ok(())
1382024-02-16T03:00:33.294ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
1392024-02-16T03:00:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1402024-02-16T03:00:33.295ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1412024-02-16T03:00:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
1422024-02-16T03:00:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1432024-02-16T03:00:33.295ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1442024-02-16T03:00:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1452024-02-16T03:00:33.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1462024-02-16T03:00:33.296ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
1472024-02-16T03:00:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1482024-02-16T03:00:33.296ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 10.977131ms iteration = 6
1492024-02-16T03:00:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1502024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
1512024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 11.443638ms iteration = 6
1522024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1532024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
1542024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
1552024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
1562024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1572024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1582024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1592024-02-16T03:00:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1602024-02-16T03:00:33.298ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 12.688599ms iteration = 26
1612024-02-16T03:00:33.298ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
1622024-02-16T03:00:33.298ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
1632024-02-16T03:00:33.298ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
1642024-02-16T03:00:33.298ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1652024-02-16T03:00:33.298ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1662024-02-16T03:00:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1672024-02-16T03:00:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
1682024-02-16T03:00:33.299ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1692024-02-16T03:00:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1702024-02-16T03:00:33.300ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1712024-02-16T03:00:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1722024-02-16T03:00:33.302ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1732024-02-16T03:00:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1742024-02-16T03:00:33.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1752024-02-16T03:00:33.303ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1762024-02-16T03:00:33.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1772024-02-16T03:00:33.303ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
1782024-02-16T03:00:33.303ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
1792024-02-16T03:00:33.303ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1802024-02-16T03:00:33.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1812024-02-16T03:00:33.305ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1822024-02-16T03:00:33.305ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
1832024-02-16T03:00:33.305ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 0 version = 2
1842024-02-16T03:00:33.305ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1852024-02-16T03:00:33.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1862024-02-16T03:00:33.307ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1872024-02-16T03:00:33.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1882024-02-16T03:00:33.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1892024-02-16T03:00:33.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1902024-02-16T03:00:33.310ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1912024-02-16T03:00:33.310ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1922024-02-16T03:00:33.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1932024-02-16T03:00:33.311ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
1942024-02-16T03:00:33.311ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
1952024-02-16T03:00:33.311ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-16 02:56:55.251295 UTC dns_group = internal generation = 1
1962024-02-16T03:00:33.311ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 26.194072ms iteration = 6
1972024-02-16T03:00:33.311ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 1 version = 2
1982024-02-16T03:00:33.312ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
1992024-02-16T03:00:33.312ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-16 02:56:55.524992 UTC dns_group = external generation = 2
2002024-02-16T03:00:33.312ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 26.428406ms iteration = 6
2012024-02-16T03:00:33.312ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2022024-02-16T03:00:33.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2032024-02-16T03:00:33.314ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2042024-02-16T03:00:33.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2052024-02-16T03:00:33.315ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2062024-02-16T03:00:33.317ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
2072024-02-16T03:00:33.317ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2082024-02-16T03:00:33.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2092024-02-16T03:00:33.318ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 32.993504ms iteration = 10
2102024-02-16T03:00:33.348ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
2112024-02-16T03:00:33.357ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2122024-02-16T03:00:33.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2132024-02-16T03:00:33.358ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2142024-02-16T03:00:33.359ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
2152024-02-16T03:00:33.359ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
2162024-02-16T03:00:33.359ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2172024-02-16T03:00:33.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2182024-02-16T03:00:33.365ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-16T02:56:33.342710Z, time_modified: 2024-02-16T02:56:33.342710Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }) }
2192024-02-16T03:00:33.365ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 79.647233ms iteration = 6
2202024-02-16T03:00:33.408ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
2212024-02-16T03:00:33.408ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "83b1c047-0f1e-46b6-8f94-3131037b30e0", "date": "Fri, 16 Feb 2024 03:00:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
2222024-02-16T03:00:33.408ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
2232024-02-16T03:00:33.410ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "803e54ac-0176-46ea-86b4-9211b0340698", "date": "Fri, 16 Feb 2024 03:00:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
2242024-02-16T03:00:33.410ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
2252024-02-16T03:00:33.412ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
2262024-02-16T03:00:33.412ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 126.492415ms iteration = 9
2272024-02-16T03:00:33.416ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
2282024-02-16T03:00:33.445ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
2292024-02-16T03:00:33.445ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "69b19a80-eab7-4da3-b892-670cf48c4e40", "date": "Fri, 16 Feb 2024 03:00:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2302024-02-16T03:00:33.446ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2312024-02-16T03:00:33.446ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a13ef894-e730-434b-be6a-3e1cd78db915", "date": "Fri, 16 Feb 2024 03:00:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2322024-02-16T03:00:33.446ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2332024-02-16T03:00:33.448ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "87663874-2e22-4e21-ba66-bee07d6d9c73", "date": "Fri, 16 Feb 2024 03:00:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2342024-02-16T03:00:33.448ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2352024-02-16T03:00:33.450ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2362024-02-16T03:00:33.450ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 158.868819ms iteration = 9
2372024-02-16T03:00:33.535ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 315 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 724936fa-bd4d-4a22-a7e5-c74aca653369 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
2382024-02-16T03:00:43.285ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
2392024-02-16T03:00:43.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2402024-02-16T03:00:43.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2412024-02-16T03:00:43.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2422024-02-16T03:00:43.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2432024-02-16T03:00:43.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2442024-02-16T03:00:43.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2452024-02-16T03:00:43.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.714111ms iteration = 27
2462024-02-16T03:00:43.535ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 181 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 3382792e-73bc-4528-a9a4-9ba571cb8af0 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
2472024-02-16T03:00:53.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
2482024-02-16T03:00:53.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2492024-02-16T03:00:53.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2502024-02-16T03:00:53.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2512024-02-16T03:00:53.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2522024-02-16T03:00:53.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2532024-02-16T03:00:53.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2542024-02-16T03:00:53.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.767675ms iteration = 28
2552024-02-16T03:00:53.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 198 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = e998fb9f-29cf-4ff5-9cba-7749594de3fc response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
2562024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
2572024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
2582024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2592024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
2602024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
2612024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
2622024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2632024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
2642024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2652024-02-16T03:01:03.286ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
2662024-02-16T03:01:03.286ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
2672024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 10.45\u{b5}s iteration = 11
2682024-02-16T03:01:03.286ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
2692024-02-16T03:01:03.286ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
2702024-02-16T03:01:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2712024-02-16T03:01:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
2722024-02-16T03:01:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2732024-02-16T03:01:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2742024-02-16T03:01:03.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2752024-02-16T03:01:03.288ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
2762024-02-16T03:01:03.288ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2772024-02-16T03:01:03.288ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
2782024-02-16T03:01:03.288ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.284058ms iteration = 11
2792024-02-16T03:01:03.289ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
2802024-02-16T03:01:03.289ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "552af1a0-44b6-4100-85e9-088a144c4143", "content-length": "1", "date": "Fri, 16 Feb 2024 03:01:02 GMT"} })
2812024-02-16T03:01:03.289ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2822024-02-16T03:01:03.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2832024-02-16T03:01:03.290ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d300c160-0f4d-4569-bd3c-a009b82e9496", "content-length": "2", "date": "Fri, 16 Feb 2024 03:01:03 GMT"} })
2842024-02-16T03:01:03.290ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 3.665221ms iteration = 11
2852024-02-16T03:01:03.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
2862024-02-16T03:01:03.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2872024-02-16T03:01:03.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2882024-02-16T03:01:03.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2892024-02-16T03:01:03.291ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.748498ms iteration = 11
2902024-02-16T03:01:03.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2912024-02-16T03:01:03.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2922024-02-16T03:01:03.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2932024-02-16T03:01:03.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.275603ms iteration = 29
2942024-02-16T03:01:03.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2952024-02-16T03:01:03.303ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2962024-02-16T03:01:03.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2972024-02-16T03:01:03.304ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2982024-02-16T03:01:03.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2992024-02-16T03:01:03.305ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3002024-02-16T03:01:03.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
3012024-02-16T03:01:03.307ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3022024-02-16T03:01:03.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3032024-02-16T03:01:03.308ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 22.275607ms iteration = 11
3042024-02-16T03:01:03.535ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 270 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 553598c2-1f6a-45a9-bd6a-43432346b876 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
3052024-02-16T03:01:13.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
3062024-02-16T03:01:13.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3072024-02-16T03:01:13.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3082024-02-16T03:01:13.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3092024-02-16T03:01:13.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3102024-02-16T03:01:13.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3112024-02-16T03:01:13.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3122024-02-16T03:01:13.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.506951ms iteration = 30
3132024-02-16T03:01:13.535ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 270 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 47ee37ac-345e-41c9-b4c9-43e84ce12dca response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
3142024-02-16T03:01:23.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
3152024-02-16T03:01:23.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3162024-02-16T03:01:23.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3172024-02-16T03:01:23.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3182024-02-16T03:01:23.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3192024-02-16T03:01:23.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3202024-02-16T03:01:23.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3212024-02-16T03:01:23.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.266873ms iteration = 31
3222024-02-16T03:01:23.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 1088 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 84f3ea40-1bc1-41a4-b65a-db513c399040 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
3232024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
3242024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
3252024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3262024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3272024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
3282024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
3292024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
3302024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3312024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
3322024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3332024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
3342024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
3352024-02-16T03:01:33.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
3362024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3372024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3382024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
3392024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3402024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
3412024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
3422024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
3432024-02-16T03:01:33.286ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
3442024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3452024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
3462024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3472024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
3482024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
3492024-02-16T03:01:33.286ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
3502024-02-16T03:01:33.286ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
3512024-02-16T03:01:33.286ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
3522024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.35\u{b5}s iteration = 7
3532024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.39\u{b5}s iteration = 12
3542024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
3552024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
3562024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3572024-02-16T03:01:33.286ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3582024-02-16T03:01:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3592024-02-16T03:01:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3602024-02-16T03:01:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
3612024-02-16T03:01:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3622024-02-16T03:01:33.288ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3632024-02-16T03:01:33.288ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3642024-02-16T03:01:33.288ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
3652024-02-16T03:01:33.288ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3662024-02-16T03:01:33.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3672024-02-16T03:01:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3682024-02-16T03:01:33.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3692024-02-16T03:01:33.289ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c7a4d808-c76b-482e-9c7e-b6ff21999700", "content-length": "1", "date": "Fri, 16 Feb 2024 03:01:32 GMT"} })
3702024-02-16T03:01:33.289ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3712024-02-16T03:01:33.289ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
3722024-02-16T03:01:33.289ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.679014ms iteration = 12
3732024-02-16T03:01:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3742024-02-16T03:01:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3752024-02-16T03:01:33.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3762024-02-16T03:01:33.290ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3772024-02-16T03:01:33.290ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a0aed8ed-1591-4cba-8f89-f97b592399d8", "content-length": "2", "date": "Fri, 16 Feb 2024 03:01:33 GMT"} })
3782024-02-16T03:01:33.290ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 3.938125ms iteration = 12
3792024-02-16T03:01:33.290ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3802024-02-16T03:01:33.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3812024-02-16T03:01:33.291ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.85088ms iteration = 12
3822024-02-16T03:01:33.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
3832024-02-16T03:01:33.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3842024-02-16T03:01:33.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Fleet result = Ok(())
3852024-02-16T03:01:33.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
3862024-02-16T03:01:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
3872024-02-16T03:01:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3882024-02-16T03:01:33.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3892024-02-16T03:01:33.294ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3902024-02-16T03:01:33.294ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3912024-02-16T03:01:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3922024-02-16T03:01:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3932024-02-16T03:01:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Fleet result = Ok(())
3942024-02-16T03:01:33.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
3952024-02-16T03:01:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
3962024-02-16T03:01:33.296ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.795201ms iteration = 32
3972024-02-16T03:01:33.296ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
3982024-02-16T03:01:33.296ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 10.940711ms iteration = 7
3992024-02-16T03:01:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4002024-02-16T03:01:33.297ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4012024-02-16T03:01:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4022024-02-16T03:01:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4032024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
4042024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 12.533286ms iteration = 7
4052024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
4062024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
4072024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
4082024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4092024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4102024-02-16T03:01:33.298ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4112024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4122024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4132024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
4142024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
4152024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
4162024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4172024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4182024-02-16T03:01:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4192024-02-16T03:01:33.300ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4202024-02-16T03:01:33.300ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4212024-02-16T03:01:33.300ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4222024-02-16T03:01:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4232024-02-16T03:01:33.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
4242024-02-16T03:01:33.304ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4252024-02-16T03:01:33.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4262024-02-16T03:01:33.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4272024-02-16T03:01:33.305ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4282024-02-16T03:01:33.305ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4292024-02-16T03:01:33.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4302024-02-16T03:01:33.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4312024-02-16T03:01:33.306ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4322024-02-16T03:01:33.306ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4332024-02-16T03:01:33.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4342024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4352024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
4362024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 0 version = 2
4372024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4382024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4392024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
4402024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
4412024-02-16T03:01:33.307ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4422024-02-16T03:01:33.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4432024-02-16T03:01:33.308ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4442024-02-16T03:01:33.308ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4452024-02-16T03:01:33.308ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4462024-02-16T03:01:33.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4472024-02-16T03:01:33.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4482024-02-16T03:01:33.312ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4492024-02-16T03:01:33.312ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4502024-02-16T03:01:33.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4512024-02-16T03:01:33.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4522024-02-16T03:01:33.313ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4532024-02-16T03:01:33.313ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-16T02:56:33.342710Z, time_modified: 2024-02-16T02:56:33.342710Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }) }
4542024-02-16T03:01:33.313ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 28.007702ms iteration = 7
4552024-02-16T03:01:33.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4562024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
4572024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
4582024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-16 02:56:55.251295 UTC dns_group = internal generation = 1
4592024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 28.748174ms iteration = 7
4602024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 1 version = 2
4612024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
4622024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-16 02:56:55.524992 UTC dns_group = external generation = 2
4632024-02-16T03:01:33.314ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 29.449805ms iteration = 7
4642024-02-16T03:01:33.315ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4652024-02-16T03:01:33.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4662024-02-16T03:01:33.317ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4672024-02-16T03:01:33.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4682024-02-16T03:01:33.318ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4692024-02-16T03:01:33.321ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
4702024-02-16T03:01:33.321ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4712024-02-16T03:01:33.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4722024-02-16T03:01:33.322ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 35.700448ms iteration = 12
4732024-02-16T03:01:33.348ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
4742024-02-16T03:01:33.360ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
4752024-02-16T03:01:33.410ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
4762024-02-16T03:01:33.410ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "c5043f29-6ef4-4ea7-89f2-f97fee2093fa", "date": "Fri, 16 Feb 2024 03:01:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
4772024-02-16T03:01:33.410ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
4782024-02-16T03:01:33.411ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6cac74ce-f11f-4232-bea2-da3b92030754", "date": "Fri, 16 Feb 2024 03:01:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
4792024-02-16T03:01:33.411ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
4802024-02-16T03:01:33.413ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
4812024-02-16T03:01:33.413ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 127.826477ms iteration = 10
4822024-02-16T03:01:33.417ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
4832024-02-16T03:01:33.448ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
4842024-02-16T03:01:33.448ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "357892c1-130f-43cb-8a72-2b67a963cf50", "date": "Fri, 16 Feb 2024 03:01:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4852024-02-16T03:01:33.448ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4862024-02-16T03:01:33.449ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4ac1cc44-34f3-4c85-98aa-e5e2849f7d20", "date": "Fri, 16 Feb 2024 03:01:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4872024-02-16T03:01:33.449ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4882024-02-16T03:01:33.450ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "14e44c28-8f57-428a-8cf9-f264acb89e33", "date": "Fri, 16 Feb 2024 03:01:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4892024-02-16T03:01:33.450ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4902024-02-16T03:01:33.451ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4912024-02-16T03:01:33.451ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 159.204774ms iteration = 10
4922024-02-16T03:01:33.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 153 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 3fdb5798-b18d-487e-8ab7-da8d8cfb674f response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
4932024-02-16T03:01:43.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
4942024-02-16T03:01:43.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4952024-02-16T03:01:43.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4962024-02-16T03:01:43.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4972024-02-16T03:01:43.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4982024-02-16T03:01:43.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4992024-02-16T03:01:43.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5002024-02-16T03:01:43.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.047549ms iteration = 33
5012024-02-16T03:01:43.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 173 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 89e36c9c-184d-4d71-a665-9ab1001e2da1 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
5022024-02-16T03:01:53.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
5032024-02-16T03:01:53.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5042024-02-16T03:01:53.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5052024-02-16T03:01:53.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5062024-02-16T03:01:53.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5072024-02-16T03:01:53.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5082024-02-16T03:01:53.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5092024-02-16T03:01:53.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.607552ms iteration = 34
5102024-02-16T03:01:53.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 733 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 40c46433-8050-4012-b464-3041e285812a response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
5112024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
5122024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
5132024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5142024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5152024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
5162024-02-16T03:02:03.287ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
5172024-02-16T03:02:03.287ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
5182024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 13.311\u{b5}s iteration = 13
5192024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
5202024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
5212024-02-16T03:02:03.287ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
5222024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5232024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
5242024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5252024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5262024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5272024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5282024-02-16T03:02:03.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5292024-02-16T03:02:03.289ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5302024-02-16T03:02:03.289ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
5312024-02-16T03:02:03.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5322024-02-16T03:02:03.290ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
5332024-02-16T03:02:03.290ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
5342024-02-16T03:02:03.290ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.01316ms iteration = 13
5352024-02-16T03:02:03.290ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "590da897-7902-472e-abf4-03dca6f356ee", "content-length": "2", "date": "Fri, 16 Feb 2024 03:02:03 GMT"} })
5362024-02-16T03:02:03.290ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 3.290614ms iteration = 13
5372024-02-16T03:02:03.290ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "57c77587-ad12-4eb1-bed9-0167194e7332", "content-length": "1", "date": "Fri, 16 Feb 2024 03:02:02 GMT"} })
5382024-02-16T03:02:03.290ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5392024-02-16T03:02:03.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5402024-02-16T03:02:03.292ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 5.041293ms iteration = 13
5412024-02-16T03:02:03.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5422024-02-16T03:02:03.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5432024-02-16T03:02:03.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5442024-02-16T03:02:03.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5452024-02-16T03:02:03.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5462024-02-16T03:02:03.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5472024-02-16T03:02:03.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5482024-02-16T03:02:03.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 7.642686ms iteration = 35
5492024-02-16T03:02:03.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5502024-02-16T03:02:03.307ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5512024-02-16T03:02:03.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5522024-02-16T03:02:03.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5532024-02-16T03:02:03.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5542024-02-16T03:02:03.310ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5552024-02-16T03:02:03.312ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
5562024-02-16T03:02:03.312ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5572024-02-16T03:02:03.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5582024-02-16T03:02:03.313ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 26.101391ms iteration = 13
5592024-02-16T03:02:03.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 246 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 5ca628e3-636f-4a93-9798-25b0a3a302b2 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
5602024-02-16T03:02:13.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
5612024-02-16T03:02:13.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5622024-02-16T03:02:13.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5632024-02-16T03:02:13.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5642024-02-16T03:02:13.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5652024-02-16T03:02:13.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5662024-02-16T03:02:13.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5672024-02-16T03:02:13.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.552728ms iteration = 36
5682024-02-16T03:02:13.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 173 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 164e01c9-88a1-4701-aef0-72f6c195315c response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
5692024-02-16T03:02:23.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
5702024-02-16T03:02:23.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5712024-02-16T03:02:23.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5722024-02-16T03:02:23.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5732024-02-16T03:02:23.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5742024-02-16T03:02:23.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5752024-02-16T03:02:23.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5762024-02-16T03:02:23.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.515058ms iteration = 37
5772024-02-16T03:02:23.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 181 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 6d44b1e6-3007-4d58-8017-3a867a335b2d response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
5782024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
5792024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
5802024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
5812024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
5822024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5832024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5842024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
5852024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
5862024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
5872024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
5882024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5892024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
5902024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
5912024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5922024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5932024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
5942024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
5952024-02-16T03:02:33.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5962024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
5972024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
5982024-02-16T03:02:33.287ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
5992024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
6002024-02-16T03:02:33.287ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
6012024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 5.81\u{b5}s iteration = 8
6022024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
6032024-02-16T03:02:33.287ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
6042024-02-16T03:02:33.287ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
6052024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6062024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 3.97\u{b5}s iteration = 14
6072024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
6082024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
6092024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6102024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
6112024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6122024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6132024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6152024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6162024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6172024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6182024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6192024-02-16T03:02:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6202024-02-16T03:02:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6212024-02-16T03:02:33.288ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6222024-02-16T03:02:33.289ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4d69a68b-8e6c-4ea7-b3de-4435f2898f02", "content-length": "1", "date": "Fri, 16 Feb 2024 03:02:32 GMT"} })
6232024-02-16T03:02:33.289ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6242024-02-16T03:02:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6252024-02-16T03:02:33.289ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
6262024-02-16T03:02:33.289ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
6272024-02-16T03:02:33.289ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.721685ms iteration = 14
6282024-02-16T03:02:33.290ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "bba73d29-2166-425b-aa1e-1daa851b4b67", "content-length": "2", "date": "Fri, 16 Feb 2024 03:02:33 GMT"} })
6292024-02-16T03:02:33.290ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.816437ms iteration = 14
6302024-02-16T03:02:33.292ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.992192ms iteration = 14
6312024-02-16T03:02:33.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6322024-02-16T03:02:33.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6332024-02-16T03:02:33.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6342024-02-16T03:02:33.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6352024-02-16T03:02:33.293ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6362024-02-16T03:02:33.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6372024-02-16T03:02:33.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6382024-02-16T03:02:33.297ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6392024-02-16T03:02:33.297ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402024-02-16T03:02:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Fleet result = Ok(())
6412024-02-16T03:02:33.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
6422024-02-16T03:02:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6432024-02-16T03:02:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Fleet result = Ok(())
6442024-02-16T03:02:33.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
6452024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
6462024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
6472024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
6482024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6492024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6502024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6512024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6522024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6532024-02-16T03:02:33.298ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
6542024-02-16T03:02:33.299ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6552024-02-16T03:02:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6562024-02-16T03:02:33.299ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
6572024-02-16T03:02:33.299ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 13.848188ms iteration = 8
6582024-02-16T03:02:33.300ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
6592024-02-16T03:02:33.300ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 13.820668ms iteration = 8
6602024-02-16T03:02:33.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6612024-02-16T03:02:33.300ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 13.070136ms iteration = 38
6622024-02-16T03:02:33.300ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6632024-02-16T03:02:33.300ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6642024-02-16T03:02:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6652024-02-16T03:02:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6662024-02-16T03:02:33.301ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
6672024-02-16T03:02:33.301ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
6682024-02-16T03:02:33.301ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
6692024-02-16T03:02:33.301ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6702024-02-16T03:02:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6712024-02-16T03:02:33.302ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
6722024-02-16T03:02:33.302ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
6732024-02-16T03:02:33.302ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
6742024-02-16T03:02:33.302ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6752024-02-16T03:02:33.302ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6762024-02-16T03:02:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6772024-02-16T03:02:33.304ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6782024-02-16T03:02:33.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
6792024-02-16T03:02:33.304ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6802024-02-16T03:02:33.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6812024-02-16T03:02:33.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6822024-02-16T03:02:33.305ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6832024-02-16T03:02:33.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6842024-02-16T03:02:33.306ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6852024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6862024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6872024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6882024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6892024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6902024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6912024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6922024-02-16T03:02:33.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6932024-02-16T03:02:33.308ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6942024-02-16T03:02:33.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6952024-02-16T03:02:33.309ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6962024-02-16T03:02:33.309ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 0 version = 2
6972024-02-16T03:02:33.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6982024-02-16T03:02:33.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6992024-02-16T03:02:33.310ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
7002024-02-16T03:02:33.310ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7012024-02-16T03:02:33.310ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7022024-02-16T03:02:33.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7032024-02-16T03:02:33.312ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
7042024-02-16T03:02:33.312ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7052024-02-16T03:02:33.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7062024-02-16T03:02:33.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7072024-02-16T03:02:33.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7082024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
7092024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7102024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7112024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
7122024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
7132024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-16 02:56:55.251295 UTC dns_group = internal generation = 1
7142024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 28.460019ms iteration = 8
7152024-02-16T03:02:33.314ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7162024-02-16T03:02:33.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7172024-02-16T03:02:33.315ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-16T02:56:33.342710Z, time_modified: 2024-02-16T02:56:33.342710Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }) }
7182024-02-16T03:02:33.315ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 29.316074ms iteration = 8
7192024-02-16T03:02:33.315ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 1 version = 2
7202024-02-16T03:02:33.315ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
7212024-02-16T03:02:33.315ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-16 02:56:55.524992 UTC dns_group = external generation = 2
7222024-02-16T03:02:33.315ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 29.337223ms iteration = 8
7232024-02-16T03:02:33.316ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7242024-02-16T03:02:33.318ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
7252024-02-16T03:02:33.318ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7262024-02-16T03:02:33.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7272024-02-16T03:02:33.320ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 32.986544ms iteration = 14
7282024-02-16T03:02:33.325ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
7292024-02-16T03:02:33.337ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
7302024-02-16T03:02:33.363ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
7312024-02-16T03:02:33.363ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "efa111cd-8888-4d5b-9ce6-9580ddffffa9", "date": "Fri, 16 Feb 2024 03:02:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
7322024-02-16T03:02:33.364ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
7332024-02-16T03:02:33.365ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "bed28fb5-99e9-4c62-9e7f-1ec551049660", "date": "Fri, 16 Feb 2024 03:02:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
7342024-02-16T03:02:33.365ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
7352024-02-16T03:02:33.367ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
7362024-02-16T03:02:33.367ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 81.825499ms iteration = 11
7372024-02-16T03:02:33.373ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
7382024-02-16T03:02:33.402ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
7392024-02-16T03:02:33.402ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "df5eba0e-2b9b-49a9-a60d-a997348b6bb0", "date": "Fri, 16 Feb 2024 03:02:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7402024-02-16T03:02:33.402ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7412024-02-16T03:02:33.403ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "c230d32f-7032-4c87-8bc0-a430048f3f72", "date": "Fri, 16 Feb 2024 03:02:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7422024-02-16T03:02:33.403ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7432024-02-16T03:02:33.405ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "0f60bb1e-3079-4f85-bbc4-10c70e5fd78e", "date": "Fri, 16 Feb 2024 03:02:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7442024-02-16T03:02:33.405ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7452024-02-16T03:02:33.407ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7462024-02-16T03:02:33.407ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 108.810284ms iteration = 11
7472024-02-16T03:02:33.536ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 173 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 8e5cc66c-3e01-4816-a6f1-bd738b0dec0b response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
7482024-02-16T03:02:43.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
7492024-02-16T03:02:43.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7502024-02-16T03:02:43.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7512024-02-16T03:02:43.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7522024-02-16T03:02:43.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7532024-02-16T03:02:43.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7542024-02-16T03:02:43.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7552024-02-16T03:02:43.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.594082ms iteration = 39
7562024-02-16T03:02:43.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 287 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = c5a778f1-5433-4566-839c-709c5836ced5 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
7572024-02-16T03:02:53.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
7582024-02-16T03:02:53.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7592024-02-16T03:02:53.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7602024-02-16T03:02:53.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7612024-02-16T03:02:53.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7622024-02-16T03:02:53.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7632024-02-16T03:02:53.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7642024-02-16T03:02:53.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.029789ms iteration = 40
7652024-02-16T03:02:53.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 1018 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 92dde47a-b455-4cf2-9e47-3fde7351a2d7 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
7662024-02-16T03:03:03.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
7672024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7682024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
7692024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7702024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
7712024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
7722024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7732024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7742024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
7752024-02-16T03:03:03.288ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
7762024-02-16T03:03:03.288ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
7772024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 10.93\u{b5}s iteration = 15
7782024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
7792024-02-16T03:03:03.288ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
7802024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7812024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7822024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7832024-02-16T03:03:03.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7842024-02-16T03:03:03.290ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7852024-02-16T03:03:03.290ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "12a52368-838a-47e8-8149-d00772e8eefb", "content-length": "1", "date": "Fri, 16 Feb 2024 03:03:02 GMT"} })
7862024-02-16T03:03:03.290ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7872024-02-16T03:03:03.290ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7882024-02-16T03:03:03.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7892024-02-16T03:03:03.290ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
7902024-02-16T03:03:03.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7912024-02-16T03:03:03.290ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "692a3a7b-9fa8-4ac9-8365-81c90a6cd869", "content-length": "2", "date": "Fri, 16 Feb 2024 03:03:03 GMT"} })
7922024-02-16T03:03:03.290ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.890408ms iteration = 15
7932024-02-16T03:03:03.291ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
7942024-02-16T03:03:03.291ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.946568ms iteration = 15
7952024-02-16T03:03:03.292ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.130038ms iteration = 15
7962024-02-16T03:03:03.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
7972024-02-16T03:03:03.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7982024-02-16T03:03:03.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7992024-02-16T03:03:03.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8002024-02-16T03:03:03.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8012024-02-16T03:03:03.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8022024-02-16T03:03:03.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 7.86673ms iteration = 41
8032024-02-16T03:03:03.295ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8042024-02-16T03:03:03.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8052024-02-16T03:03:03.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8062024-02-16T03:03:03.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8072024-02-16T03:03:03.310ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8082024-02-16T03:03:03.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8092024-02-16T03:03:03.311ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8102024-02-16T03:03:03.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
8112024-02-16T03:03:03.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8122024-02-16T03:03:03.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8132024-02-16T03:03:03.314ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 26.958774ms iteration = 15
8142024-02-16T03:03:03.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 160 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 6023a749-03c4-4f8e-861a-8c4eed68814b response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
8152024-02-16T03:03:13.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
8162024-02-16T03:03:13.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8172024-02-16T03:03:13.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8182024-02-16T03:03:13.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8192024-02-16T03:03:13.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8202024-02-16T03:03:13.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8212024-02-16T03:03:13.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8222024-02-16T03:03:13.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.600532ms iteration = 42
8232024-02-16T03:03:13.538ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 1182 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = fc89420e-1403-46e7-940b-864ee3dc8e36 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
8242024-02-16T03:03:20.886ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::1:0]:60516
8252024-02-16T03:03:20.919ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:374 id = c698e8ce-2e84-48e7-ae07-086e0f5433d0
8262024-02-16T03:03:20.925ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = c698e8ce-2e84-48e7-ae07-086e0f5433d0 uri = http://[fd00:1122:3344:101::d]:12223/producers
8272024-02-16T03:03:20.926ZDEBGnexus: client response oximeter-collector = c698e8ce-2e84-48e7-ae07-086e0f5433d0 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "fa038f90-c041-43df-8ed2-92669672a4b2", "date": "Fri, 16 Feb 2024 03:03:20 GMT"} })
8282024-02-16T03:03:20.926ZINFOnexus: assigned collector to new producer collector_id = c698e8ce-2e84-48e7-ae07-086e0f5433d0 file = nexus/src/app/oximeter.rs:193 producer_id = ae46dd0e-145d-46e9-945e-d85ca59da5e4
8292024-02-16T03:03:20.927ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 40288 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:60516 req_id = e48a3c34-0113-41de-87b3-dbd6a89f5629 response_code = 204 uri = /metrics/producers
8302024-02-16T03:03:23.288ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
8312024-02-16T03:03:23.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8322024-02-16T03:03:23.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8332024-02-16T03:03:23.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8342024-02-16T03:03:23.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8352024-02-16T03:03:23.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8362024-02-16T03:03:23.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8372024-02-16T03:03:23.298ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.905994ms iteration = 43
8382024-02-16T03:03:23.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 270 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 10d41aff-593d-496c-8628-9e4c1a996de1 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
8392024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
8402024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
8412024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
8422024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
8432024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8442024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
8452024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
8462024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
8472024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
8482024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
8492024-02-16T03:03:33.286ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
8502024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8512024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8522024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8532024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8542024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
8552024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8562024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
8572024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8582024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
8592024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
8602024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
8612024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8622024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
8632024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8642024-02-16T03:03:33.287ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
8652024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
8662024-02-16T03:03:33.288ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
8672024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8682024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
8692024-02-16T03:03:33.288ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
8702024-02-16T03:03:33.288ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
8712024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
8722024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 5.07\u{b5}s iteration = 16
8732024-02-16T03:03:33.288ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
8742024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.93\u{b5}s iteration = 9
8752024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8762024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
8772024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8782024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8792024-02-16T03:03:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8802024-02-16T03:03:33.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8812024-02-16T03:03:33.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8822024-02-16T03:03:33.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8832024-02-16T03:03:33.290ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8842024-02-16T03:03:33.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8852024-02-16T03:03:33.290ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
8862024-02-16T03:03:33.290ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.629393ms iteration = 16
8872024-02-16T03:03:33.290ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
8882024-02-16T03:03:33.290ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8892024-02-16T03:03:33.290ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "72740681-0bfd-4855-a624-ed5cc0b1ac29", "content-length": "2", "date": "Fri, 16 Feb 2024 03:03:33 GMT"} })
8902024-02-16T03:03:33.290ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.914958ms iteration = 16
8912024-02-16T03:03:33.290ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8922024-02-16T03:03:33.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8932024-02-16T03:03:33.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8942024-02-16T03:03:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Fleet result = Ok(())
8952024-02-16T03:03:33.294ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
8962024-02-16T03:03:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
8972024-02-16T03:03:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8982024-02-16T03:03:33.295ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
8992024-02-16T03:03:33.295ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9002024-02-16T03:03:33.295ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9012024-02-16T03:03:33.295ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ddb54f52-07ea-4e25-94e7-1deba791ba1f", "content-length": "1", "date": "Fri, 16 Feb 2024 03:03:32 GMT"} })
9022024-02-16T03:03:33.295ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9032024-02-16T03:03:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9042024-02-16T03:03:33.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9052024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9062024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9072024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Fleet result = Ok(())
9082024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
9092024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9102024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
9112024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9122024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9132024-02-16T03:03:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9142024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
9152024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
9162024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
9172024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9182024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
9192024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 10.410022ms iteration = 9
9202024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9212024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9222024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9232024-02-16T03:03:33.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9242024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 10.112137ms iteration = 44
9252024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9262024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 10.726507ms iteration = 16
9272024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9282024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
9292024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 12.029058ms iteration = 9
9302024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9312024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
9322024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
9332024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
9342024-02-16T03:03:33.298ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9352024-02-16T03:03:33.299ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9362024-02-16T03:03:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9372024-02-16T03:03:33.300ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
9382024-02-16T03:03:33.300ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9392024-02-16T03:03:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9402024-02-16T03:03:33.302ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9412024-02-16T03:03:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9422024-02-16T03:03:33.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
9432024-02-16T03:03:33.303ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9442024-02-16T03:03:33.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9452024-02-16T03:03:33.304ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
9462024-02-16T03:03:33.304ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 0 version = 2
9472024-02-16T03:03:33.304ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9482024-02-16T03:03:33.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9492024-02-16T03:03:33.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9502024-02-16T03:03:33.304ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9512024-02-16T03:03:33.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9522024-02-16T03:03:33.305ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9532024-02-16T03:03:33.305ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9542024-02-16T03:03:33.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9552024-02-16T03:03:33.306ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
9562024-02-16T03:03:33.306ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
9572024-02-16T03:03:33.306ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9582024-02-16T03:03:33.306ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9592024-02-16T03:03:33.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9602024-02-16T03:03:33.308ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9612024-02-16T03:03:33.308ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9622024-02-16T03:03:33.308ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9632024-02-16T03:03:33.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9642024-02-16T03:03:33.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9652024-02-16T03:03:33.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9662024-02-16T03:03:33.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9672024-02-16T03:03:33.310ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9682024-02-16T03:03:33.310ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 1 version = 2
9692024-02-16T03:03:33.310ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
9702024-02-16T03:03:33.310ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-16 02:56:55.524992 UTC dns_group = external generation = 2
9712024-02-16T03:03:33.310ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.582139ms iteration = 9
9722024-02-16T03:03:33.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9732024-02-16T03:03:33.311ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9742024-02-16T03:03:33.311ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9752024-02-16T03:03:33.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9762024-02-16T03:03:33.312ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9772024-02-16T03:03:33.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9782024-02-16T03:03:33.313ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
9792024-02-16T03:03:33.313ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
9802024-02-16T03:03:33.313ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-16 02:56:55.251295 UTC dns_group = internal generation = 1
9812024-02-16T03:03:33.313ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 26.493767ms iteration = 9
9822024-02-16T03:03:33.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9832024-02-16T03:03:33.314ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-16T02:56:33.342710Z, time_modified: 2024-02-16T02:56:33.342710Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }) } file = nexus/src/app/background/external_endpoints.rs:101
9842024-02-16T03:03:33.314ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 27.631085ms iteration = 9
9852024-02-16T03:03:33.315ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
9862024-02-16T03:03:33.315ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9872024-02-16T03:03:33.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9882024-02-16T03:03:33.317ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 29.320193ms iteration = 16
9892024-02-16T03:03:33.341ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
9902024-02-16T03:03:33.367ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
9912024-02-16T03:03:33.398ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
9922024-02-16T03:03:33.399ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "e192ea47-ffb0-4f59-97c0-33bfae360b00", "date": "Fri, 16 Feb 2024 03:03:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9932024-02-16T03:03:33.399ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9942024-02-16T03:03:33.401ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d7d78845-7dc5-488e-a2e4-a7e7adcd9b99", "date": "Fri, 16 Feb 2024 03:03:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9952024-02-16T03:03:33.401ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9962024-02-16T03:03:33.402ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
9972024-02-16T03:03:33.402ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 115.563295ms iteration = 12
9982024-02-16T03:03:33.412ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
9992024-02-16T03:03:33.443ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
10002024-02-16T03:03:33.443ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d75ec30a-f181-4b39-aa4b-287e92e71886", "date": "Fri, 16 Feb 2024 03:03:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
10012024-02-16T03:03:33.443ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
10022024-02-16T03:03:33.444ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a1f20742-fd9e-4841-9c68-af07c8bac0a7", "date": "Fri, 16 Feb 2024 03:03:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
10032024-02-16T03:03:33.444ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
10042024-02-16T03:03:33.446ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "814dff90-fa67-4e70-859f-70dfc16fab24", "date": "Fri, 16 Feb 2024 03:03:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
10052024-02-16T03:03:33.446ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
10062024-02-16T03:03:33.448ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
10072024-02-16T03:03:33.448ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 148.102561ms iteration = 12
10082024-02-16T03:03:33.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 234 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = e5a5daa8-44cf-4039-9b14-c7061a7a9f1b response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
10092024-02-16T03:03:43.287ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
10102024-02-16T03:03:43.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10112024-02-16T03:03:43.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10122024-02-16T03:03:43.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10132024-02-16T03:03:43.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10142024-02-16T03:03:43.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10152024-02-16T03:03:43.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10162024-02-16T03:03:43.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.602769ms iteration = 45
10172024-02-16T03:03:43.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 240 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 31ec221d-f048-48c1-bd1e-e251404337ab response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
10182024-02-16T03:03:53.288ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
10192024-02-16T03:03:53.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10202024-02-16T03:03:53.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10212024-02-16T03:03:53.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10222024-02-16T03:03:53.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10232024-02-16T03:03:53.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10242024-02-16T03:03:53.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10252024-02-16T03:03:53.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.979089ms iteration = 46
10262024-02-16T03:03:53.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 160 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 13c8ae9b-448e-4f47-84a3-c77cda31f837 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
10272024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
10282024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
10292024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10302024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10312024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
10322024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10332024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
10342024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
10352024-02-16T03:04:03.288ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
10362024-02-16T03:04:03.288ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
10372024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 8.39\u{b5}s iteration = 17
10382024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10392024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
10402024-02-16T03:04:03.288ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
10412024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
10422024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10432024-02-16T03:04:03.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10442024-02-16T03:04:03.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10452024-02-16T03:04:03.290ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
10462024-02-16T03:04:03.290ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0b212051-2a6c-4272-a7e9-896a757bf38b", "content-length": "2", "date": "Fri, 16 Feb 2024 03:04:02 GMT"} })
10472024-02-16T03:04:03.290ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10482024-02-16T03:04:03.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10492024-02-16T03:04:03.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10502024-02-16T03:04:03.291ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
10512024-02-16T03:04:03.291ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.700515ms iteration = 17
10522024-02-16T03:04:03.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10532024-02-16T03:04:03.291ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
10542024-02-16T03:04:03.292ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "68771375-e95a-447b-9440-f6bb13b923a6", "content-length": "2", "date": "Fri, 16 Feb 2024 03:04:03 GMT"} })
10552024-02-16T03:04:03.292ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 4.171779ms iteration = 17
10562024-02-16T03:04:03.292ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.395483ms iteration = 17
10572024-02-16T03:04:03.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10582024-02-16T03:04:03.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10592024-02-16T03:04:03.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10602024-02-16T03:04:03.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
10612024-02-16T03:04:03.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10622024-02-16T03:04:03.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10632024-02-16T03:04:03.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 7.382562ms iteration = 47
10642024-02-16T03:04:03.296ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10652024-02-16T03:04:03.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10662024-02-16T03:04:03.306ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10672024-02-16T03:04:03.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10682024-02-16T03:04:03.307ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10692024-02-16T03:04:03.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10702024-02-16T03:04:03.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10712024-02-16T03:04:03.310ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
10722024-02-16T03:04:03.310ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10732024-02-16T03:04:03.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10742024-02-16T03:04:03.312ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 23.831453ms iteration = 17
10752024-02-16T03:04:03.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 181 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = a12108e3-8af1-479e-8195-fd5294886cbc response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
10762024-02-16T03:04:13.289ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
10772024-02-16T03:04:13.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10782024-02-16T03:04:13.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10792024-02-16T03:04:13.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10802024-02-16T03:04:13.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10812024-02-16T03:04:13.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10822024-02-16T03:04:13.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10832024-02-16T03:04:13.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.319844ms iteration = 48
10842024-02-16T03:04:13.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 169 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 129948c6-49fa-45cb-ae9e-e793f996625b response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
10852024-02-16T03:04:23.288ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
10862024-02-16T03:04:23.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10872024-02-16T03:04:23.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10882024-02-16T03:04:23.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10892024-02-16T03:04:23.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10902024-02-16T03:04:23.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10912024-02-16T03:04:23.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10922024-02-16T03:04:23.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.411666ms iteration = 49
10932024-02-16T03:04:23.538ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 165 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = b13ab62a-9aa9-4baf-930c-d1babd8f65d9 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
10942024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
10952024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
10962024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
10972024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10982024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
10992024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
11002024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
11012024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
11022024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11032024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
11042024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
11052024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11062024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
11072024-02-16T03:04:33.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11082024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
11092024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11102024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11112024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
11122024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
11132024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
11142024-02-16T03:04:33.288ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
11152024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11162024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
11172024-02-16T03:04:33.288ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
11182024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.421\u{b5}s iteration = 10
11192024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
11202024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
11212024-02-16T03:04:33.288ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
11222024-02-16T03:04:33.288ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
11232024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11242024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.61\u{b5}s iteration = 18
11252024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
11262024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11272024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
11282024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
11292024-02-16T03:04:33.288ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11302024-02-16T03:04:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
11312024-02-16T03:04:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11322024-02-16T03:04:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11332024-02-16T03:04:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11342024-02-16T03:04:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11352024-02-16T03:04:33.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11362024-02-16T03:04:33.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11372024-02-16T03:04:33.290ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11382024-02-16T03:04:33.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11392024-02-16T03:04:33.290ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11402024-02-16T03:04:33.291ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "201d6e80-6299-4294-a202-8f687a266306", "content-length": "2", "date": "Fri, 16 Feb 2024 03:04:32 GMT"} })
11412024-02-16T03:04:33.291ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11422024-02-16T03:04:33.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11432024-02-16T03:04:33.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11442024-02-16T03:04:33.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11452024-02-16T03:04:33.291ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
11462024-02-16T03:04:33.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11472024-02-16T03:04:33.292ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3fe6c042-ba8b-4936-8ef5-72389a7015fc", "content-length": "2", "date": "Fri, 16 Feb 2024 03:04:33 GMT"} })
11482024-02-16T03:04:33.293ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 4.327401ms iteration = 18
11492024-02-16T03:04:33.293ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.448893ms iteration = 18
11502024-02-16T03:04:33.293ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
11512024-02-16T03:04:33.293ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.681297ms iteration = 18
11522024-02-16T03:04:33.293ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
11532024-02-16T03:04:33.293ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11542024-02-16T03:04:33.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11552024-02-16T03:04:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Fleet result = Ok(())
11562024-02-16T03:04:33.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_external dns_group = external roles = RoleSet { roles: {} }
11572024-02-16T03:04:33.295ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11582024-02-16T03:04:33.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_external dns_group = external resource = Database result = Ok(())
11592024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
11602024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11612024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Fleet result = Ok(())
11622024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): roles background_task = dns_servers_internal dns_group = internal roles = RoleSet { roles: {} }
11632024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11642024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_servers_internal dns_group = internal resource = Database result = Ok(())
11652024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11662024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11672024-02-16T03:04:33.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11682024-02-16T03:04:33.297ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
11692024-02-16T03:04:33.297ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 10.162257ms iteration = 10
11702024-02-16T03:04:33.297ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11712024-02-16T03:04:33.297ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
11722024-02-16T03:04:33.297ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
11732024-02-16T03:04:33.297ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
11742024-02-16T03:04:33.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11752024-02-16T03:04:33.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11762024-02-16T03:04:33.298ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11772024-02-16T03:04:33.298ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.904693ms iteration = 50
11782024-02-16T03:04:33.298ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
11792024-02-16T03:04:33.298ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 11.215185ms iteration = 10
11802024-02-16T03:04:33.299ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11812024-02-16T03:04:33.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
11822024-02-16T03:04:33.300ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11832024-02-16T03:04:33.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11842024-02-16T03:04:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
11852024-02-16T03:04:33.301ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11862024-02-16T03:04:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11872024-02-16T03:04:33.301ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
11882024-02-16T03:04:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11892024-02-16T03:04:33.301ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11902024-02-16T03:04:33.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11912024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
11922024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
11932024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
11942024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11952024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11962024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
11972024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11982024-02-16T03:04:33.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11992024-02-16T03:04:33.305ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12002024-02-16T03:04:33.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12012024-02-16T03:04:33.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12022024-02-16T03:04:33.309ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12032024-02-16T03:04:33.310ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
12042024-02-16T03:04:33.310ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 0 version = 2
12052024-02-16T03:04:33.310ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12062024-02-16T03:04:33.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12072024-02-16T03:04:33.311ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
12082024-02-16T03:04:33.311ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12092024-02-16T03:04:33.312ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12102024-02-16T03:04:33.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12112024-02-16T03:04:33.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
12122024-02-16T03:04:33.313ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12132024-02-16T03:04:33.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12142024-02-16T03:04:33.314ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
12152024-02-16T03:04:33.314ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12162024-02-16T03:04:33.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12172024-02-16T03:04:33.316ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
12182024-02-16T03:04:33.316ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
12192024-02-16T03:04:33.316ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12202024-02-16T03:04:33.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12212024-02-16T03:04:33.317ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = ae5cdc48-b87b-4cb6-8f77-00ffb45d0c52 dns_zone_name = oxide.test found_so_far = 1 version = 2
12222024-02-16T03:04:33.317ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
12232024-02-16T03:04:33.317ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-16 02:56:55.524992 UTC dns_group = external generation = 2
12242024-02-16T03:04:33.317ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 30.197828ms iteration = 10
12252024-02-16T03:04:33.318ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12262024-02-16T03:04:33.320ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
12272024-02-16T03:04:33.320ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12282024-02-16T03:04:33.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12292024-02-16T03:04:33.320ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-16T02:56:33.342710Z, time_modified: 2024-02-16T02:56:33.342710Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 4f587a78-ade9-4425-a242-08fc8b642fb3, db_silo: Silo { identity: SiloIdentity { id: 4f587a78-ade9-4425-a242-08fc8b642fb3, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-16T02:56:55.502618Z, time_modified: 2024-02-16T02:56:55.502618Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f8d6424acc1c3a24918f77ff5da19781fd3dbb962208c758468de01082670a5a" }] }) }
12302024-02-16T03:04:33.320ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 33.266598ms iteration = 10
12312024-02-16T03:04:33.320ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12322024-02-16T03:04:33.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12332024-02-16T03:04:33.322ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12342024-02-16T03:04:33.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12352024-02-16T03:04:33.322ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 5e0f478e-5809-46c0-9aa6-9d12723a80b7 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
12362024-02-16T03:04:33.322ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
12372024-02-16T03:04:33.322ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-16 02:56:55.251295 UTC dns_group = internal generation = 1
12382024-02-16T03:04:33.322ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 35.436754ms iteration = 10
12392024-02-16T03:04:33.323ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12402024-02-16T03:04:33.325ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b2a61df6-1b19-4116-8bd9-77a3c76d28b6, lookup_type: ById(b2a61df6-1b19-4116-8bd9-77a3c76d28b6) } result = Ok(())
12412024-02-16T03:04:33.325ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12422024-02-16T03:04:33.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12432024-02-16T03:04:33.327ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 38.405553ms iteration = 18
12442024-02-16T03:04:33.330ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
12452024-02-16T03:04:33.343ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
12462024-02-16T03:04:33.371ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
12472024-02-16T03:04:33.371ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a4f9eae8-83cc-48da-a49f-d57b550e178e", "date": "Fri, 16 Feb 2024 03:04:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
12482024-02-16T03:04:33.371ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
12492024-02-16T03:04:33.372ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "28003d80-8a1c-47fb-9d8f-e09d46f5f04e", "date": "Fri, 16 Feb 2024 03:04:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
12502024-02-16T03:04:33.372ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
12512024-02-16T03:04:33.375ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
12522024-02-16T03:04:33.375ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 87.763546ms iteration = 13
12532024-02-16T03:04:33.383ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
12542024-02-16T03:04:33.413ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
12552024-02-16T03:04:33.413ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f98712c3-c179-4c56-aa53-e404e43e0936", "date": "Fri, 16 Feb 2024 03:04:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
12562024-02-16T03:04:33.413ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
12572024-02-16T03:04:33.414ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "865b6411-1f6f-4b50-94b7-789056ce2100", "date": "Fri, 16 Feb 2024 03:04:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
12582024-02-16T03:04:33.414ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
12592024-02-16T03:04:33.415ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "a9ac9a9d-7661-4da2-8b4b-a6e3cea1b954", "date": "Fri, 16 Feb 2024 03:04:33 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
12602024-02-16T03:04:33.415ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
12612024-02-16T03:04:33.415ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
12622024-02-16T03:04:33.416ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 114.717071ms iteration = 13
12632024-02-16T03:04:33.537ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 207 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = c7933556-ca8e-4f86-bcac-ed74b0deff68 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c
12642024-02-16T03:04:43.289ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
12652024-02-16T03:04:43.289ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12662024-02-16T03:04:43.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12672024-02-16T03:04:43.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12682024-02-16T03:04:43.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12692024-02-16T03:04:43.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12702024-02-16T03:04:43.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12712024-02-16T03:04:43.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.560138ms iteration = 51
12722024-02-16T03:04:43.539ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 816 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47677 req_id = 6dc70b3d-5545-48e9-be6f-e4bc89b26707 response_code = 200 uri = /metrics/collect/decd40e2-201e-4efd-bfc7-6fd7b053f29c