Max level shown:
12024-02-15T00:15:02.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
22024-02-15T00:15:02.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32024-02-15T00:15:02.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42024-02-15T00:15:02.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52024-02-15T00:15:02.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62024-02-15T00:15:02.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72024-02-15T00:15:02.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82024-02-15T00:15:02.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.217424ms iteration = 34
92024-02-15T00:15:02.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 187 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 79f4deb1-bfa8-490d-b56f-a156f537e168 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
102024-02-15T00:15:12.115ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
112024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
122024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
132024-02-15T00:15:12.116ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
142024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
152024-02-15T00:15:12.116ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
162024-02-15T00:15:12.116ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
172024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.79\u{b5}s iteration = 13
182024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
192024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
202024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
212024-02-15T00:15:12.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
222024-02-15T00:15:12.118ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
232024-02-15T00:15:12.118ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
242024-02-15T00:15:12.119ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1c772cc0-3fec-42bf-acb9-e22b39d921c3", "content-length": "2", "date": "Thu, 15 Feb 2024 00:15:11 GMT"} })
252024-02-15T00:15:12.119ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
262024-02-15T00:15:12.119ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
272024-02-15T00:15:12.119ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.359167ms iteration = 13
282024-02-15T00:15:12.119ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "74ea0008-ca0e-41f3-bb37-5a51386c83c8", "content-length": "2", "date": "Thu, 15 Feb 2024 00:15:12 GMT"} })
292024-02-15T00:15:12.119ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 3.409739ms iteration = 13
302024-02-15T00:15:12.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
312024-02-15T00:15:12.120ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.840976ms iteration = 13
322024-02-15T00:15:12.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
332024-02-15T00:15:12.136ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
342024-02-15T00:15:12.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
352024-02-15T00:15:12.136ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
362024-02-15T00:15:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
372024-02-15T00:15:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
382024-02-15T00:15:12.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
392024-02-15T00:15:12.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
402024-02-15T00:15:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
412024-02-15T00:15:12.141ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
422024-02-15T00:15:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
432024-02-15T00:15:12.141ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
442024-02-15T00:15:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
452024-02-15T00:15:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
462024-02-15T00:15:12.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.51246ms iteration = 35
472024-02-15T00:15:12.142ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
482024-02-15T00:15:12.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
492024-02-15T00:15:12.153ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
502024-02-15T00:15:12.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
512024-02-15T00:15:12.154ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
522024-02-15T00:15:12.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
532024-02-15T00:15:12.156ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
542024-02-15T00:15:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
552024-02-15T00:15:12.158ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
562024-02-15T00:15:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
572024-02-15T00:15:12.159ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 23.336945ms iteration = 13
582024-02-15T00:15:12.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 201 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 63aef665-964d-438a-b52e-4f46cfc64664 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
592024-02-15T00:15:22.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
602024-02-15T00:15:22.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
612024-02-15T00:15:22.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
622024-02-15T00:15:22.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
632024-02-15T00:15:22.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
642024-02-15T00:15:22.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
652024-02-15T00:15:22.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
662024-02-15T00:15:22.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.447239ms iteration = 36
672024-02-15T00:15:22.393ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 260 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 1caa8de4-1e66-4407-a3c1-ea5687871f6e response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
682024-02-15T00:15:32.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
692024-02-15T00:15:32.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
702024-02-15T00:15:32.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
712024-02-15T00:15:32.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
722024-02-15T00:15:32.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
732024-02-15T00:15:32.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
742024-02-15T00:15:32.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
752024-02-15T00:15:32.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.854017ms iteration = 37
762024-02-15T00:15:32.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 183 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 57d17e88-8225-4055-a502-44d0fbeede2b response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
772024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
782024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
792024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
802024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
812024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
822024-02-15T00:15:42.115ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
832024-02-15T00:15:42.115ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
842024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 17.65\u{b5}s iteration = 14
852024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
862024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
872024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
882024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
892024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
902024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
912024-02-15T00:15:42.115ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
922024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
932024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
942024-02-15T00:15:42.116ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
952024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
962024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
972024-02-15T00:15:42.116ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
982024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.4\u{b5}s iteration = 8
992024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
1002024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
1012024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
1022024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1032024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1042024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1052024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1062024-02-15T00:15:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1072024-02-15T00:15:42.116ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5e4f48e8-b987-45c0-8c5b-523ef542f945.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a3944496-e534-42b6-afe3-78830d746249.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 1708, tv_nsec: 555367778 } })
1082024-02-15T00:15:42.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1092024-02-15T00:15:42.118ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1999ec12-789c-40c8-83be-a68bad0fad69.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 1708, tv_nsec: 556629463 } })
1102024-02-15T00:15:42.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1112024-02-15T00:15:42.118ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1122024-02-15T00:15:42.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1132024-02-15T00:15:42.118ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
1142024-02-15T00:15:42.118ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1152024-02-15T00:15:42.119ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "6f703e5a-81af-4677-903e-627a79e87227", "content-length": "2", "date": "Thu, 15 Feb 2024 00:15:42 GMT"} })
1162024-02-15T00:15:42.119ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 3.315527ms iteration = 14
1172024-02-15T00:15:42.119ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1182024-02-15T00:15:42.120ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1192024-02-15T00:15:42.120ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
1202024-02-15T00:15:42.120ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.396968ms iteration = 14
1212024-02-15T00:15:42.120ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1222024-02-15T00:15:42.120ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "03505686-16e1-44f1-b18a-d045c1928d47", "content-length": "2", "date": "Thu, 15 Feb 2024 00:15:41 GMT"} })
1232024-02-15T00:15:42.120ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1242024-02-15T00:15:42.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1252024-02-15T00:15:42.121ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1262024-02-15T00:15:42.121ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1272024-02-15T00:15:42.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1282024-02-15T00:15:42.122ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
1292024-02-15T00:15:42.122ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 6.641392ms iteration = 8
1302024-02-15T00:15:42.122ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 6.99686ms iteration = 14
1312024-02-15T00:15:42.123ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1322024-02-15T00:15:42.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1332024-02-15T00:15:42.123ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
1342024-02-15T00:15:42.123ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 7.913668ms iteration = 8
1352024-02-15T00:15:42.123ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1362024-02-15T00:15:42.124ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1372024-02-15T00:15:42.124ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1382024-02-15T00:15:42.124ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
1392024-02-15T00:15:42.124ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
1402024-02-15T00:15:42.124ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
1412024-02-15T00:15:42.124ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1422024-02-15T00:15:42.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1432024-02-15T00:15:42.125ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
1442024-02-15T00:15:42.125ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
1452024-02-15T00:15:42.125ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
1462024-02-15T00:15:42.125ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1472024-02-15T00:15:42.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1482024-02-15T00:15:42.126ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1492024-02-15T00:15:42.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1502024-02-15T00:15:42.128ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
1512024-02-15T00:15:42.128ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1522024-02-15T00:15:42.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1532024-02-15T00:15:42.130ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1542024-02-15T00:15:42.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1552024-02-15T00:15:42.130ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1562024-02-15T00:15:42.130ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1572024-02-15T00:15:42.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1582024-02-15T00:15:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1592024-02-15T00:15:42.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1602024-02-15T00:15:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1612024-02-15T00:15:42.131ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
1622024-02-15T00:15:42.131ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 0 version = 2
1632024-02-15T00:15:42.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1642024-02-15T00:15:42.131ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1652024-02-15T00:15:42.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1662024-02-15T00:15:42.132ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
1672024-02-15T00:15:42.132ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
1682024-02-15T00:15:42.132ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1692024-02-15T00:15:42.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1702024-02-15T00:15:42.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1712024-02-15T00:15:42.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1722024-02-15T00:15:42.133ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1732024-02-15T00:15:42.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1742024-02-15T00:15:42.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1752024-02-15T00:15:42.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
1762024-02-15T00:15:42.135ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
1772024-02-15T00:15:42.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1782024-02-15T00:15:42.135ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1792024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1802024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1812024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1822024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1832024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1842024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1852024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1862024-02-15T00:15:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1872024-02-15T00:15:42.137ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 1 version = 2
1882024-02-15T00:15:42.137ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
1892024-02-15T00:15:42.137ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 00:10:06.332957 UTC dns_group = external generation = 2
1902024-02-15T00:15:42.137ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 21.351675ms iteration = 8
1912024-02-15T00:15:42.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1922024-02-15T00:15:42.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1932024-02-15T00:15:42.138ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
1942024-02-15T00:15:42.138ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
1952024-02-15T00:15:42.138ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 00:10:05.994647 UTC dns_group = internal generation = 1
1962024-02-15T00:15:42.138ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 22.350796ms iteration = 8
1972024-02-15T00:15:42.139ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T00:09:42.183399Z, time_modified: 2024-02-15T00:09:42.183399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }) }
1982024-02-15T00:15:42.139ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 23.835285ms iteration = 8
1992024-02-15T00:15:42.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
2002024-02-15T00:15:42.140ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2012024-02-15T00:15:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2022024-02-15T00:15:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2032024-02-15T00:15:42.141ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2042024-02-15T00:15:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2052024-02-15T00:15:42.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.821106ms iteration = 38
2062024-02-15T00:15:42.142ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2072024-02-15T00:15:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2082024-02-15T00:15:42.155ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2092024-02-15T00:15:42.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2102024-02-15T00:15:42.157ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2112024-02-15T00:15:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2122024-02-15T00:15:42.159ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2132024-02-15T00:15:42.161ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
2142024-02-15T00:15:42.161ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2152024-02-15T00:15:42.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2162024-02-15T00:15:42.162ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 26.55235ms iteration = 14
2172024-02-15T00:15:42.173ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
2182024-02-15T00:15:42.188ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
2192024-02-15T00:15:42.234ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
2202024-02-15T00:15:42.234ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d43a38f2-37a9-4220-b800-17330aec814e", "date": "Thu, 15 Feb 2024 00:15:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2212024-02-15T00:15:42.234ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2222024-02-15T00:15:42.235ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "0bdba140-f761-4728-8503-e29bd2ddff12", "date": "Thu, 15 Feb 2024 00:15:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2232024-02-15T00:15:42.235ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2242024-02-15T00:15:42.236ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
2252024-02-15T00:15:42.236ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 121.208647ms iteration = 10
2262024-02-15T00:15:42.243ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
2272024-02-15T00:15:42.273ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
2282024-02-15T00:15:42.274ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "42bd6b48-9c26-4d8b-818a-c9a7f5979ed0", "date": "Thu, 15 Feb 2024 00:15:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2292024-02-15T00:15:42.274ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2302024-02-15T00:15:42.275ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "1e36ea4a-9418-4b69-bd09-5a8538b84e96", "date": "Thu, 15 Feb 2024 00:15:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2312024-02-15T00:15:42.275ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2322024-02-15T00:15:42.277ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "515e54c3-3816-4620-88f7-8e91944385a7", "date": "Thu, 15 Feb 2024 00:15:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2332024-02-15T00:15:42.277ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2342024-02-15T00:15:42.279ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
2352024-02-15T00:15:42.279ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 163.594492ms iteration = 10
2362024-02-15T00:15:42.393ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 193 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = fdfa61aa-bae0-4136-a8b6-05f5ac84103f response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
2372024-02-15T00:15:52.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
2382024-02-15T00:15:52.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2392024-02-15T00:15:52.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2402024-02-15T00:15:52.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2412024-02-15T00:15:52.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2422024-02-15T00:15:52.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2432024-02-15T00:15:52.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2442024-02-15T00:15:52.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.848717ms iteration = 39
2452024-02-15T00:15:52.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 158 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 9794be65-f5f7-4a11-99ab-4f1c320a14c7 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
2462024-02-15T00:16:02.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
2472024-02-15T00:16:02.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2482024-02-15T00:16:02.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2492024-02-15T00:16:02.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2502024-02-15T00:16:02.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2512024-02-15T00:16:02.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2522024-02-15T00:16:02.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2532024-02-15T00:16:02.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.378297ms iteration = 40
2542024-02-15T00:16:02.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 169 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = f5d2b54e-36da-4040-96ba-f185f2d2b0b8 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
2552024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
2562024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2572024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
2582024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
2592024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
2602024-02-15T00:16:12.116ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
2612024-02-15T00:16:12.116ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
2622024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 15.09\u{b5}s iteration = 15
2632024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
2642024-02-15T00:16:12.116ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
2652024-02-15T00:16:12.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2662024-02-15T00:16:12.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
2672024-02-15T00:16:12.118ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
2682024-02-15T00:16:12.118ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
2692024-02-15T00:16:12.119ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "21a909a1-6e3c-48bc-83df-dff3beefb91f", "content-length": "2", "date": "Thu, 15 Feb 2024 00:16:11 GMT"} })
2702024-02-15T00:16:12.119ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2712024-02-15T00:16:12.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2722024-02-15T00:16:12.119ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
2732024-02-15T00:16:12.119ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.449079ms iteration = 15
2742024-02-15T00:16:12.120ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e03cbaad-57e0-48f4-941b-32628060590c", "content-length": "2", "date": "Thu, 15 Feb 2024 00:16:12 GMT"} })
2752024-02-15T00:16:12.120ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 3.994969ms iteration = 15
2762024-02-15T00:16:12.120ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.446038ms iteration = 15
2772024-02-15T00:16:12.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
2782024-02-15T00:16:12.135ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
2792024-02-15T00:16:12.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2802024-02-15T00:16:12.135ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2812024-02-15T00:16:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2822024-02-15T00:16:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2832024-02-15T00:16:12.139ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2842024-02-15T00:16:12.139ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2852024-02-15T00:16:12.143ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2862024-02-15T00:16:12.143ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2872024-02-15T00:16:12.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2882024-02-15T00:16:12.143ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
2892024-02-15T00:16:12.143ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2902024-02-15T00:16:12.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2912024-02-15T00:16:12.144ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.111392ms iteration = 41
2922024-02-15T00:16:12.145ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2932024-02-15T00:16:12.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2942024-02-15T00:16:12.158ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2952024-02-15T00:16:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2962024-02-15T00:16:12.160ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2972024-02-15T00:16:12.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2982024-02-15T00:16:12.161ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2992024-02-15T00:16:12.163ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
3002024-02-15T00:16:12.163ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3012024-02-15T00:16:12.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3022024-02-15T00:16:12.164ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 28.866566ms iteration = 15
3032024-02-15T00:16:12.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 182 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 2f295f43-de64-45f9-b01f-4da2ef6607e9 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
3042024-02-15T00:16:22.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
3052024-02-15T00:16:22.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3062024-02-15T00:16:22.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3072024-02-15T00:16:22.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3082024-02-15T00:16:22.139ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3092024-02-15T00:16:22.139ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3102024-02-15T00:16:22.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3112024-02-15T00:16:22.141ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.612512ms iteration = 42
3122024-02-15T00:16:22.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 161 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 178a801a-3c94-44f1-ae2b-8604e471a5f1 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
3132024-02-15T00:16:32.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
3142024-02-15T00:16:32.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3152024-02-15T00:16:32.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3162024-02-15T00:16:32.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3172024-02-15T00:16:32.142ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3182024-02-15T00:16:32.142ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3192024-02-15T00:16:32.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3202024-02-15T00:16:32.145ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 9.408157ms iteration = 43
3212024-02-15T00:16:32.393ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 263 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = aed38dc8-5260-4af5-9224-5cf9ba18377b response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
3222024-02-15T00:16:37.511ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::1:0]:45936
3232024-02-15T00:16:37.543ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:374 id = a0570de6-fbf4-4073-9e84-a6e01264b6a9
3242024-02-15T00:16:37.552ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = a0570de6-fbf4-4073-9e84-a6e01264b6a9 uri = http://[fd00:1122:3344:101::d]:12223/producers
3252024-02-15T00:16:37.553ZDEBGnexus: client response oximeter-collector = a0570de6-fbf4-4073-9e84-a6e01264b6a9 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d8ceca6a-acb8-4b53-a168-785df8e6bbe8", "date": "Thu, 15 Feb 2024 00:16:37 GMT"} })
3262024-02-15T00:16:37.553ZINFOnexus: assigned collector to new producer collector_id = a0570de6-fbf4-4073-9e84-a6e01264b6a9 file = nexus/src/app/oximeter.rs:193 producer_id = 94fccb41-16a0-4df0-bce6-a17b14d4be05
3272024-02-15T00:16:37.554ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 42576 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:45936 req_id = 6d8483f4-91b6-4c8d-b5e6-4cb88a42a115 response_code = 204 uri = /metrics/producers
3282024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
3292024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3302024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
3312024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
3322024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
3332024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3342024-02-15T00:16:42.115ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
3352024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.18\u{b5}s iteration = 9
3362024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
3372024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
3382024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
3392024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3402024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
3412024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3422024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3432024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
3442024-02-15T00:16:42.115ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
3452024-02-15T00:16:42.115ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
3462024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.12\u{b5}s iteration = 16
3472024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
3482024-02-15T00:16:42.115ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
3492024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
3502024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
3512024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
3522024-02-15T00:16:42.115ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3532024-02-15T00:16:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3542024-02-15T00:16:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
3552024-02-15T00:16:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3562024-02-15T00:16:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3572024-02-15T00:16:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3582024-02-15T00:16:42.116ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1999ec12-789c-40c8-83be-a68bad0fad69.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 1768, tv_nsec: 554773186 } })
3592024-02-15T00:16:42.116ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5e4f48e8-b987-45c0-8c5b-523ef542f945.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a3944496-e534-42b6-afe3-78830d746249.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 1768, tv_nsec: 554946600 } })
3602024-02-15T00:16:42.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3612024-02-15T00:16:42.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3622024-02-15T00:16:42.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3632024-02-15T00:16:42.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3642024-02-15T00:16:42.118ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3652024-02-15T00:16:42.119ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3662024-02-15T00:16:42.119ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3672024-02-15T00:16:42.119ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
3682024-02-15T00:16:42.120ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3692024-02-15T00:16:42.120ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "991b21ed-a1ea-457c-a7aa-c83029496bd3", "content-length": "2", "date": "Thu, 15 Feb 2024 00:16:41 GMT"} })
3702024-02-15T00:16:42.120ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3712024-02-15T00:16:42.120ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
3722024-02-15T00:16:42.120ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 4.710864ms iteration = 9
3732024-02-15T00:16:42.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3742024-02-15T00:16:42.120ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
3752024-02-15T00:16:42.120ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.979619ms iteration = 16
3762024-02-15T00:16:42.121ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4df4e2a1-7b10-4d6f-a488-af5f1d7c70c6", "content-length": "2", "date": "Thu, 15 Feb 2024 00:16:42 GMT"} })
3772024-02-15T00:16:42.121ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 6.136222ms iteration = 16
3782024-02-15T00:16:42.122ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
3792024-02-15T00:16:42.122ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 6.457639ms iteration = 9
3802024-02-15T00:16:42.122ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 6.53604ms iteration = 16
3812024-02-15T00:16:42.124ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3822024-02-15T00:16:42.124ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3832024-02-15T00:16:42.124ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3842024-02-15T00:16:42.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3852024-02-15T00:16:42.125ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3862024-02-15T00:16:42.125ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
3872024-02-15T00:16:42.125ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
3882024-02-15T00:16:42.126ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
3892024-02-15T00:16:42.126ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3902024-02-15T00:16:42.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3912024-02-15T00:16:42.126ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3922024-02-15T00:16:42.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3932024-02-15T00:16:42.127ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
3942024-02-15T00:16:42.127ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
3952024-02-15T00:16:42.127ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
3962024-02-15T00:16:42.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3972024-02-15T00:16:42.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3982024-02-15T00:16:42.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3992024-02-15T00:16:42.128ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4002024-02-15T00:16:42.128ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4012024-02-15T00:16:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4022024-02-15T00:16:42.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4032024-02-15T00:16:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4042024-02-15T00:16:42.132ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
4052024-02-15T00:16:42.132ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
4062024-02-15T00:16:42.132ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4072024-02-15T00:16:42.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4082024-02-15T00:16:42.133ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
4092024-02-15T00:16:42.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4102024-02-15T00:16:42.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4112024-02-15T00:16:42.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4122024-02-15T00:16:42.134ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4132024-02-15T00:16:42.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4142024-02-15T00:16:42.134ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4152024-02-15T00:16:42.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4162024-02-15T00:16:42.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4172024-02-15T00:16:42.135ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
4182024-02-15T00:16:42.135ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 0 version = 2
4192024-02-15T00:16:42.135ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4202024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
4212024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4222024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
4232024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4242024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4252024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4262024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4272024-02-15T00:16:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4282024-02-15T00:16:42.137ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4292024-02-15T00:16:42.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4302024-02-15T00:16:42.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4312024-02-15T00:16:42.138ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4322024-02-15T00:16:42.138ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4332024-02-15T00:16:42.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4342024-02-15T00:16:42.139ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
4352024-02-15T00:16:42.139ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
4362024-02-15T00:16:42.139ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 00:10:05.994647 UTC dns_group = internal generation = 1
4372024-02-15T00:16:42.139ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 24.262833ms iteration = 9
4382024-02-15T00:16:42.140ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4392024-02-15T00:16:42.140ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4402024-02-15T00:16:42.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4412024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4422024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4432024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4442024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4452024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4462024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4472024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4482024-02-15T00:16:42.142ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4492024-02-15T00:16:42.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4502024-02-15T00:16:42.143ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 7.733054ms iteration = 44
4512024-02-15T00:16:42.143ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4522024-02-15T00:16:42.144ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 1 version = 2
4532024-02-15T00:16:42.144ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
4542024-02-15T00:16:42.144ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 00:10:06.332957 UTC dns_group = external generation = 2
4552024-02-15T00:16:42.144ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 28.469988ms iteration = 9
4562024-02-15T00:16:42.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4572024-02-15T00:16:42.145ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T00:09:42.183399Z, time_modified: 2024-02-15T00:09:42.183399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }) } file = nexus/src/app/background/external_endpoints.rs:101
4582024-02-15T00:16:42.145ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 29.975688ms iteration = 9
4592024-02-15T00:16:42.154ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4602024-02-15T00:16:42.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4612024-02-15T00:16:42.156ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4622024-02-15T00:16:42.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4632024-02-15T00:16:42.157ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4642024-02-15T00:16:42.161ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
4652024-02-15T00:16:42.161ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4662024-02-15T00:16:42.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4672024-02-15T00:16:42.162ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 25.940647ms iteration = 16
4682024-02-15T00:16:42.185ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
4692024-02-15T00:16:42.189ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
4702024-02-15T00:16:42.247ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
4712024-02-15T00:16:42.247ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "addc2d4f-ae0a-42de-8279-f6840a40c561", "date": "Thu, 15 Feb 2024 00:16:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4722024-02-15T00:16:42.247ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4732024-02-15T00:16:42.249ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "9b2ab357-7d5f-48ad-b0a5-92a0f8a21ba8", "date": "Thu, 15 Feb 2024 00:16:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4742024-02-15T00:16:42.249ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4752024-02-15T00:16:42.250ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
4762024-02-15T00:16:42.250ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4772024-02-15T00:16:42.251ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 135.333458ms iteration = 11
4782024-02-15T00:16:42.281ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
4792024-02-15T00:16:42.282ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "aa85eb4a-403b-4597-90b4-ffc848dd2eaf", "date": "Thu, 15 Feb 2024 00:16:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4802024-02-15T00:16:42.283ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4812024-02-15T00:16:42.284ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "21cc4def-f338-4ac8-bacc-dd140ad8b7aa", "date": "Thu, 15 Feb 2024 00:16:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4822024-02-15T00:16:42.284ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4832024-02-15T00:16:42.285ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "368ee6ee-12ef-4eee-8c47-cee7e401bdbd", "date": "Thu, 15 Feb 2024 00:16:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4842024-02-15T00:16:42.285ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4852024-02-15T00:16:42.287ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
4862024-02-15T00:16:42.287ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 172.119911ms iteration = 11
4872024-02-15T00:16:42.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 157 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 64b1b6a7-dd14-4bf0-a7ff-236024e6b3a1 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
4882024-02-15T00:16:52.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
4892024-02-15T00:16:52.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4902024-02-15T00:16:52.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4912024-02-15T00:16:52.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4922024-02-15T00:16:52.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4932024-02-15T00:16:52.141ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4942024-02-15T00:16:52.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4952024-02-15T00:16:52.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.310116ms iteration = 45
4962024-02-15T00:16:52.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 156 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = a48e2230-1c57-4459-a93c-46ab255db7c5 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
4972024-02-15T00:17:02.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
4982024-02-15T00:17:02.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4992024-02-15T00:17:02.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5002024-02-15T00:17:02.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5012024-02-15T00:17:02.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5022024-02-15T00:17:02.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5032024-02-15T00:17:02.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5042024-02-15T00:17:02.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.387327ms iteration = 46
5052024-02-15T00:17:02.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 237 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 42c96ba6-d5f5-45a1-9234-e8be3fd12d05 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
5062024-02-15T00:17:12.115ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
5072024-02-15T00:17:12.115ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5082024-02-15T00:17:12.115ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
5092024-02-15T00:17:12.115ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
5102024-02-15T00:17:12.115ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
5112024-02-15T00:17:12.115ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
5122024-02-15T00:17:12.115ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5132024-02-15T00:17:12.115ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
5142024-02-15T00:17:12.115ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
5152024-02-15T00:17:12.115ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 12.6\u{b5}s iteration = 17
5162024-02-15T00:17:12.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5172024-02-15T00:17:12.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5182024-02-15T00:17:12.117ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
5192024-02-15T00:17:12.118ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9f06f5fd-805b-4ba6-a6b4-ba9af3f6b9b4", "content-length": "2", "date": "Thu, 15 Feb 2024 00:17:12 GMT"} })
5202024-02-15T00:17:12.118ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.421348ms iteration = 17
5212024-02-15T00:17:12.118ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
5222024-02-15T00:17:12.118ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "49f06748-8e15-41af-a712-68aab75b558c", "content-length": "2", "date": "Thu, 15 Feb 2024 00:17:11 GMT"} })
5232024-02-15T00:17:12.118ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5242024-02-15T00:17:12.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5252024-02-15T00:17:12.119ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
5262024-02-15T00:17:12.119ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.55234ms iteration = 17
5272024-02-15T00:17:12.119ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 4.065111ms iteration = 17
5282024-02-15T00:17:12.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
5292024-02-15T00:17:12.135ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
5302024-02-15T00:17:12.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5312024-02-15T00:17:12.136ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5322024-02-15T00:17:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5332024-02-15T00:17:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5342024-02-15T00:17:12.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5352024-02-15T00:17:12.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5362024-02-15T00:17:12.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5372024-02-15T00:17:12.140ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5382024-02-15T00:17:12.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5392024-02-15T00:17:12.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5402024-02-15T00:17:12.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5412024-02-15T00:17:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5422024-02-15T00:17:12.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.244984ms iteration = 47
5432024-02-15T00:17:12.142ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5442024-02-15T00:17:12.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5452024-02-15T00:17:12.152ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5462024-02-15T00:17:12.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5472024-02-15T00:17:12.154ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5482024-02-15T00:17:12.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5492024-02-15T00:17:12.155ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5502024-02-15T00:17:12.157ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
5512024-02-15T00:17:12.157ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5522024-02-15T00:17:12.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5532024-02-15T00:17:12.158ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 22.641661ms iteration = 17
5542024-02-15T00:17:12.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 151 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 9976613c-bf23-41dd-8c28-9f2a8dde725e response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
5552024-02-15T00:17:22.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
5562024-02-15T00:17:22.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5572024-02-15T00:17:22.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5582024-02-15T00:17:22.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5592024-02-15T00:17:22.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5602024-02-15T00:17:22.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5612024-02-15T00:17:22.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5622024-02-15T00:17:22.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.950769ms iteration = 48
5632024-02-15T00:17:22.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 260 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 0ab83f7b-443f-4a19-bb2b-0ce261cff7e2 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
5642024-02-15T00:17:32.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
5652024-02-15T00:17:32.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5662024-02-15T00:17:32.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5672024-02-15T00:17:32.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5682024-02-15T00:17:32.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5692024-02-15T00:17:32.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5702024-02-15T00:17:32.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5712024-02-15T00:17:32.141ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 5.770305ms iteration = 49
5722024-02-15T00:17:32.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 223 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 807ae001-d461-4962-92dd-7ec65b41c239 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
5732024-02-15T00:17:42.115ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
5742024-02-15T00:17:42.115ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5752024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
5762024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
5772024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5782024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
5792024-02-15T00:17:42.116ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
5802024-02-15T00:17:42.116ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
5812024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 13.06\u{b5}s iteration = 18
5822024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
5832024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
5842024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
5852024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
5862024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5872024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
5882024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
5892024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
5902024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5912024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
5922024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
5932024-02-15T00:17:42.116ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
5942024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5952024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
5962024-02-15T00:17:42.116ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
5972024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.89\u{b5}s iteration = 10
5982024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5992024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6002024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6012024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6022024-02-15T00:17:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6032024-02-15T00:17:42.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6042024-02-15T00:17:42.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6052024-02-15T00:17:42.118ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6062024-02-15T00:17:42.118ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6072024-02-15T00:17:42.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6082024-02-15T00:17:42.119ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1999ec12-789c-40c8-83be-a68bad0fad69.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 1828, tv_nsec: 557679824 } })
6092024-02-15T00:17:42.119ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5e4f48e8-b987-45c0-8c5b-523ef542f945.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a3944496-e534-42b6-afe3-78830d746249.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 1828, tv_nsec: 557746705 } })
6102024-02-15T00:17:42.119ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6112024-02-15T00:17:42.119ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6122024-02-15T00:17:42.120ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
6132024-02-15T00:17:42.120ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1b28f6e8-153a-4e42-9dfd-ebbd6747e9b0", "content-length": "2", "date": "Thu, 15 Feb 2024 00:17:41 GMT"} })
6142024-02-15T00:17:42.120ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6152024-02-15T00:17:42.120ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6162024-02-15T00:17:42.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6172024-02-15T00:17:42.120ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
6182024-02-15T00:17:42.120ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.753224ms iteration = 18
6192024-02-15T00:17:42.121ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f454287c-53b8-4050-b2fa-5890a4dc598b", "content-length": "2", "date": "Thu, 15 Feb 2024 00:17:42 GMT"} })
6202024-02-15T00:17:42.121ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 5.702333ms iteration = 18
6212024-02-15T00:17:42.121ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6222024-02-15T00:17:42.121ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6232024-02-15T00:17:42.121ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 6.084951ms iteration = 18
6242024-02-15T00:17:42.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6252024-02-15T00:17:42.123ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6262024-02-15T00:17:42.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6272024-02-15T00:17:42.124ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6282024-02-15T00:17:42.124ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6292024-02-15T00:17:42.124ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
6302024-02-15T00:17:42.124ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6312024-02-15T00:17:42.124ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 8.754845ms iteration = 10
6322024-02-15T00:17:42.125ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
6332024-02-15T00:17:42.125ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
6342024-02-15T00:17:42.125ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
6352024-02-15T00:17:42.125ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6362024-02-15T00:17:42.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6372024-02-15T00:17:42.126ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
6382024-02-15T00:17:42.126ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
6392024-02-15T00:17:42.126ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
6402024-02-15T00:17:42.126ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6412024-02-15T00:17:42.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6422024-02-15T00:17:42.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6432024-02-15T00:17:42.127ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
6442024-02-15T00:17:42.127ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 11.363597ms iteration = 10
6452024-02-15T00:17:42.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6462024-02-15T00:17:42.128ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
6472024-02-15T00:17:42.128ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6482024-02-15T00:17:42.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6492024-02-15T00:17:42.129ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6502024-02-15T00:17:42.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6512024-02-15T00:17:42.130ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6522024-02-15T00:17:42.132ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6532024-02-15T00:17:42.132ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6542024-02-15T00:17:42.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6552024-02-15T00:17:42.132ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6562024-02-15T00:17:42.132ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6572024-02-15T00:17:42.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6582024-02-15T00:17:42.133ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6592024-02-15T00:17:42.133ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 0 version = 2
6602024-02-15T00:17:42.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6612024-02-15T00:17:42.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6622024-02-15T00:17:42.133ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6632024-02-15T00:17:42.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6642024-02-15T00:17:42.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6652024-02-15T00:17:42.134ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6662024-02-15T00:17:42.134ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6672024-02-15T00:17:42.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6682024-02-15T00:17:42.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6692024-02-15T00:17:42.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6702024-02-15T00:17:42.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
6712024-02-15T00:17:42.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6722024-02-15T00:17:42.135ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6732024-02-15T00:17:42.135ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
6742024-02-15T00:17:42.135ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6752024-02-15T00:17:42.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6762024-02-15T00:17:42.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6772024-02-15T00:17:42.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6782024-02-15T00:17:42.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6792024-02-15T00:17:42.139ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T00:09:42.183399Z, time_modified: 2024-02-15T00:09:42.183399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }) }
6802024-02-15T00:17:42.139ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 23.271214ms iteration = 10
6812024-02-15T00:17:42.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6822024-02-15T00:17:42.140ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6832024-02-15T00:17:42.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6842024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 1 version = 2
6852024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
6862024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 00:10:06.332957 UTC dns_group = external generation = 2
6872024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 25.418197ms iteration = 10
6882024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
6892024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6902024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6912024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6922024-02-15T00:17:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6932024-02-15T00:17:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6942024-02-15T00:17:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6952024-02-15T00:17:42.142ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6962024-02-15T00:17:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6972024-02-15T00:17:42.143ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6982024-02-15T00:17:42.143ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
6992024-02-15T00:17:42.143ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
7002024-02-15T00:17:42.143ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 00:10:05.994647 UTC dns_group = internal generation = 1
7012024-02-15T00:17:42.143ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 27.475368ms iteration = 10
7022024-02-15T00:17:42.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7032024-02-15T00:17:42.143ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 8.358407ms iteration = 50
7042024-02-15T00:17:42.153ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7052024-02-15T00:17:42.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7062024-02-15T00:17:42.155ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7072024-02-15T00:17:42.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7082024-02-15T00:17:42.156ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7092024-02-15T00:17:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
7102024-02-15T00:17:42.158ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7112024-02-15T00:17:42.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7122024-02-15T00:17:42.159ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 24.128911ms iteration = 18
7132024-02-15T00:17:42.179ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
7142024-02-15T00:17:42.179ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
7152024-02-15T00:17:42.238ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
7162024-02-15T00:17:42.239ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
7172024-02-15T00:17:42.240ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "88320ac1-632a-4f75-85e7-9be4c5d29436", "date": "Thu, 15 Feb 2024 00:17:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7182024-02-15T00:17:42.240ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7192024-02-15T00:17:42.241ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d6469136-e6b3-41c6-8458-31257843c939", "date": "Thu, 15 Feb 2024 00:17:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7202024-02-15T00:17:42.241ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7212024-02-15T00:17:42.242ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7222024-02-15T00:17:42.242ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 126.772078ms iteration = 12
7232024-02-15T00:17:42.268ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
7242024-02-15T00:17:42.269ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "808cb2c9-1948-4920-aca3-769c4e389c90", "date": "Thu, 15 Feb 2024 00:17:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7252024-02-15T00:17:42.269ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7262024-02-15T00:17:42.270ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d3725ccf-c544-4096-9b3f-a80d748b1c76", "date": "Thu, 15 Feb 2024 00:17:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7272024-02-15T00:17:42.270ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7282024-02-15T00:17:42.272ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "11e10dd8-2873-48e7-816a-751293cdda75", "date": "Thu, 15 Feb 2024 00:17:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7292024-02-15T00:17:42.272ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7302024-02-15T00:17:42.274ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7312024-02-15T00:17:42.274ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 158.150473ms iteration = 12
7322024-02-15T00:17:42.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 150 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 89c754a7-6532-43c3-a515-4c74bf99fa62 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
7332024-02-15T00:17:52.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
7342024-02-15T00:17:52.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7352024-02-15T00:17:52.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7362024-02-15T00:17:52.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7372024-02-15T00:17:52.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7382024-02-15T00:17:52.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7392024-02-15T00:17:52.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7402024-02-15T00:17:52.141ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.01326ms iteration = 51
7412024-02-15T00:17:52.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 181 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 7c9f70e1-91bb-4d76-91e8-420560289e2d response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
7422024-02-15T00:18:02.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
7432024-02-15T00:18:02.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7442024-02-15T00:18:02.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7452024-02-15T00:18:02.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7462024-02-15T00:18:02.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7472024-02-15T00:18:02.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7482024-02-15T00:18:02.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7492024-02-15T00:18:02.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.233424ms iteration = 52
7502024-02-15T00:18:02.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 254 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = da911ec1-4547-4c10-8a23-d15eecf366bb response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
7512024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
7522024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7532024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
7542024-02-15T00:18:12.115ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
7552024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
7562024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
7572024-02-15T00:18:12.115ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
7582024-02-15T00:18:12.115ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
7592024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7602024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 15.17\u{b5}s iteration = 19
7612024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7622024-02-15T00:18:12.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7632024-02-15T00:18:12.117ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7642024-02-15T00:18:12.117ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e47720-064a-4e72-a292-27bdd634431e", "content-length": "2", "date": "Thu, 15 Feb 2024 00:18:12 GMT"} })
7652024-02-15T00:18:12.117ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7662024-02-15T00:18:12.117ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
7672024-02-15T00:18:12.117ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "53a48abe-1270-4017-87e3-827214697bad", "content-length": "2", "date": "Thu, 15 Feb 2024 00:18:12 GMT"} })
7682024-02-15T00:18:12.117ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 2.232224ms iteration = 19
7692024-02-15T00:18:12.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7702024-02-15T00:18:12.118ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
7712024-02-15T00:18:12.118ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.142893ms iteration = 19
7722024-02-15T00:18:12.119ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 3.602801ms iteration = 19
7732024-02-15T00:18:12.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
7742024-02-15T00:18:12.136ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
7752024-02-15T00:18:12.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7762024-02-15T00:18:12.136ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7772024-02-15T00:18:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7782024-02-15T00:18:12.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7792024-02-15T00:18:12.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7802024-02-15T00:18:12.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7812024-02-15T00:18:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
7822024-02-15T00:18:12.141ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7832024-02-15T00:18:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7842024-02-15T00:18:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7852024-02-15T00:18:12.141ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7862024-02-15T00:18:12.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7872024-02-15T00:18:12.142ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7882024-02-15T00:18:12.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.419408ms iteration = 53
7892024-02-15T00:18:12.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7902024-02-15T00:18:12.152ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7912024-02-15T00:18:12.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7922024-02-15T00:18:12.154ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7932024-02-15T00:18:12.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7942024-02-15T00:18:12.155ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7952024-02-15T00:18:12.157ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
7962024-02-15T00:18:12.157ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7972024-02-15T00:18:12.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7982024-02-15T00:18:12.159ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 22.812065ms iteration = 19
7992024-02-15T00:18:12.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 247 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 9502abfd-8c0b-40f6-b05a-a12029a851a5 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
8002024-02-15T00:18:22.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
8012024-02-15T00:18:22.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8022024-02-15T00:18:22.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8032024-02-15T00:18:22.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8042024-02-15T00:18:22.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8052024-02-15T00:18:22.141ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8062024-02-15T00:18:22.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8072024-02-15T00:18:22.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.48923ms iteration = 54
8082024-02-15T00:18:22.393ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 938 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 8615edbb-455f-4d3d-9765-fa95eff14cbe response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
8092024-02-15T00:18:32.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
8102024-02-15T00:18:32.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8112024-02-15T00:18:32.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8122024-02-15T00:18:32.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8132024-02-15T00:18:32.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8142024-02-15T00:18:32.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8152024-02-15T00:18:32.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8162024-02-15T00:18:32.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.555911ms iteration = 55
8172024-02-15T00:18:32.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 174 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = c09367e9-4b3f-44f7-8b7a-8d7f7f9b0042 response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
8182024-02-15T00:18:42.115ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
8192024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8202024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
8212024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
8222024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
8232024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8242024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
8252024-02-15T00:18:42.116ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
8262024-02-15T00:18:42.116ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
8272024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 24.38\u{b5}s iteration = 20
8282024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
8292024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
8302024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
8312024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
8322024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
8332024-02-15T00:18:42.116ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
8342024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
8352024-02-15T00:18:42.116ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:46 reason = no blueprint
8362024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
8372024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
8382024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.411\u{b5}s iteration = 11
8392024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
8402024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8412024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8422024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
8432024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8442024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8452024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8462024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
8472024-02-15T00:18:42.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8482024-02-15T00:18:42.117ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8492024-02-15T00:18:42.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8502024-02-15T00:18:42.118ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("5e4f48e8-b987-45c0-8c5b-523ef542f945.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a3944496-e534-42b6-afe3-78830d746249.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("eaa4434d-a6bd-48f6-beb2-151404cd7796.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 1888, tv_nsec: 557462260 } })
8512024-02-15T00:18:42.118ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8522024-02-15T00:18:42.118ZDEBGnexus: client request MgAdminClient = [fd00:1122:3344:101::2]:4676 body = None method = GET uri = http://[fd00:1122:3344:101::2]:4676/bfd/peers
8532024-02-15T00:18:42.119ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1999ec12-789c-40c8-83be-a68bad0fad69.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("1db31f1b-e3f9-47c5-a2c6-e5eba50ff1ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 1888, tv_nsec: 557709045 } })
8542024-02-15T00:18:42.119ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8552024-02-15T00:18:42.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8562024-02-15T00:18:42.119ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8572024-02-15T00:18:42.119ZDEBGnexus (DpdClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
8582024-02-15T00:18:42.120ZDEBGnexus: client response MgAdminClient = [fd00:1122:3344:101::2]:4676 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(4676), path: "/bfd/peers", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7f21822e-eced-46f0-8b2b-c0ae1172dbb7", "content-length": "2", "date": "Thu, 15 Feb 2024 00:18:42 GMT"} })
8592024-02-15T00:18:42.120ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 4.170143ms iteration = 20
8602024-02-15T00:18:42.120ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
8612024-02-15T00:18:42.120ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.157063ms iteration = 20
8622024-02-15T00:18:42.120ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8632024-02-15T00:18:42.121ZDEBGnexus (DpdClient): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7985dff6-914b-4008-94c7-59995ca970b4", "content-length": "2", "date": "Thu, 15 Feb 2024 00:18:42 GMT"} })
8642024-02-15T00:18:42.121ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8652024-02-15T00:18:42.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8662024-02-15T00:18:42.122ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8672024-02-15T00:18:42.122ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8682024-02-15T00:18:42.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8692024-02-15T00:18:42.123ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 7.324836ms iteration = 20
8702024-02-15T00:18:42.124ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8712024-02-15T00:18:42.124ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8722024-02-15T00:18:42.124ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8732024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8742024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
8752024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
8762024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
8772024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8782024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
8792024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 9.807706ms iteration = 11
8802024-02-15T00:18:42.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8812024-02-15T00:18:42.126ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
8822024-02-15T00:18:42.126ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
8832024-02-15T00:18:42.126ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
8842024-02-15T00:18:42.126ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8852024-02-15T00:18:42.126ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8862024-02-15T00:18:42.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8872024-02-15T00:18:42.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8882024-02-15T00:18:42.127ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
8892024-02-15T00:18:42.127ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 11.103651ms iteration = 11
8902024-02-15T00:18:42.127ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8912024-02-15T00:18:42.130ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
8922024-02-15T00:18:42.130ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8932024-02-15T00:18:42.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8942024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8952024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8962024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8972024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8982024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8992024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9002024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9012024-02-15T00:18:42.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9022024-02-15T00:18:42.132ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
9032024-02-15T00:18:42.132ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 0 version = 2
9042024-02-15T00:18:42.132ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9052024-02-15T00:18:42.133ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9062024-02-15T00:18:42.133ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
9072024-02-15T00:18:42.133ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
9082024-02-15T00:18:42.133ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9092024-02-15T00:18:42.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9102024-02-15T00:18:42.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9112024-02-15T00:18:42.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9122024-02-15T00:18:42.134ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9132024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9142024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9152024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
9162024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9172024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9182024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
9192024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9202024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9212024-02-15T00:18:42.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9222024-02-15T00:18:42.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9232024-02-15T00:18:42.137ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242024-02-15T00:18:42.138ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9252024-02-15T00:18:42.138ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9262024-02-15T00:18:42.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9272024-02-15T00:18:42.139ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9282024-02-15T00:18:42.139ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9292024-02-15T00:18:42.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9302024-02-15T00:18:42.139ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 365ff672-d73e-42b8-b377-c5db60450337 dns_zone_name = oxide.test found_so_far = 1 version = 2
9312024-02-15T00:18:42.139ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
9322024-02-15T00:18:42.139ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-02-15 00:10:06.332957 UTC dns_group = external generation = 2
9332024-02-15T00:18:42.139ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 23.496898ms iteration = 11
9342024-02-15T00:18:42.140ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-02-15T00:09:42.183399Z, time_modified: 2024-02-15T00:09:42.183399Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, db_silo: Silo { identity: SiloIdentity { id: 7edfbd27-e261-4bbd-b0a4-b4fb0910a699, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-02-15T00:10:06.310840Z, time_modified: 2024-02-15T00:10:06.310840Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "2a797b0372b3c4c1682379147cfd3057d99dbf561387c185f5a92bfa4fb536ab" }] }) }
9352024-02-15T00:18:42.140ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 24.283364ms iteration = 11
9362024-02-15T00:18:42.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9372024-02-15T00:18:42.140ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9382024-02-15T00:18:42.140ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 9a882901-9470-4d7b-82ae-4f3f626e82c5 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
9392024-02-15T00:18:42.141ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
9402024-02-15T00:18:42.141ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-02-15 00:10:05.994647 UTC dns_group = internal generation = 1
9412024-02-15T00:18:42.141ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 25.10431ms iteration = 11
9422024-02-15T00:18:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9432024-02-15T00:18:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9442024-02-15T00:18:42.141ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9452024-02-15T00:18:42.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9462024-02-15T00:18:42.142ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9472024-02-15T00:18:42.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9482024-02-15T00:18:42.143ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 7.619202ms iteration = 56
9492024-02-15T00:18:42.152ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9502024-02-15T00:18:42.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9512024-02-15T00:18:42.153ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9522024-02-15T00:18:42.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9532024-02-15T00:18:42.154ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9542024-02-15T00:18:42.157ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: c76945c0-20aa-4b87-817a-70fea9ec6d5a, lookup_type: ById(c76945c0-20aa-4b87-817a-70fea9ec6d5a) } result = Ok(())
9552024-02-15T00:18:42.157ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9562024-02-15T00:18:42.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9572024-02-15T00:18:42.158ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 23.036079ms iteration = 20
9582024-02-15T00:18:42.179ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
9592024-02-15T00:18:42.184ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
9602024-02-15T00:18:42.236ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
9612024-02-15T00:18:42.237ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "53a99be2-e607-4adb-aad6-612c12e4643b", "date": "Thu, 15 Feb 2024 00:18:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9622024-02-15T00:18:42.237ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9632024-02-15T00:18:42.238ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "1965f4dd-e8ce-420c-98a7-185bef95b3ac", "date": "Thu, 15 Feb 2024 00:18:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9642024-02-15T00:18:42.238ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9652024-02-15T00:18:42.239ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9662024-02-15T00:18:42.239ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 123.458242ms iteration = 13
9672024-02-15T00:18:42.239ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
9682024-02-15T00:18:42.269ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
9692024-02-15T00:18:42.269ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "c61c21d9-e325-45f0-9a89-ad424b6aa9c7", "date": "Thu, 15 Feb 2024 00:18:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9702024-02-15T00:18:42.269ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9712024-02-15T00:18:42.270ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "49283e7d-df4c-4269-beba-e593f827d7b2", "date": "Thu, 15 Feb 2024 00:18:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9722024-02-15T00:18:42.270ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9732024-02-15T00:18:42.272ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "23526553-8104-461a-8f17-6450ea9362b9", "date": "Thu, 15 Feb 2024 00:18:42 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9742024-02-15T00:18:42.272ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9752024-02-15T00:18:42.274ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
9762024-02-15T00:18:42.274ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 158.406958ms iteration = 13
9772024-02-15T00:18:42.391ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 168 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = 29fb3f52-6960-4d8d-9185-cfa85c88189b response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3
9782024-02-15T00:18:52.136ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
9792024-02-15T00:18:52.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9802024-02-15T00:18:52.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9812024-02-15T00:18:52.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9822024-02-15T00:18:52.140ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9832024-02-15T00:18:52.140ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9842024-02-15T00:18:52.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9852024-02-15T00:18:52.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 6.693094ms iteration = 57
9862024-02-15T00:18:52.392ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/711a749/dropshot/src/server.rs:849 latency_us = 249 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49383 req_id = b720abab-1746-48ea-9763-68dc7599bb7c response_code = 200 uri = /metrics/collect/42ed2c6e-2acb-4dbc-9b39-365d99de97d3