Max level shown:
12025-03-20T17:15:02.178ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
22025-03-20T17:15:02.178ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
32025-03-20T17:15:02.180ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42025-03-20T17:15:02.181ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 508.158514ms iteration = 23 status = null
52025-03-20T17:15:02.184ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
62025-03-20T17:15:02.184ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72025-03-20T17:15:02.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
82025-03-20T17:15:02.186ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92025-03-20T17:15:02.189ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
102025-03-20T17:15:02.189ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
112025-03-20T17:15:02.189ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
122025-03-20T17:15:02.200ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 526.89751ms iteration = 23 status = {}
132025-03-20T17:15:02.206ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
142025-03-20T17:15:02.207ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "0b206aa6-2af4-4e61-a402-9a6c0fca5f6c", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
152025-03-20T17:15:02.207ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
162025-03-20T17:15:02.209ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "92aa9b25-5f87-4162-8719-41ec5e001590", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
172025-03-20T17:15:02.209ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
182025-03-20T17:15:02.210ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e5a9d3b6-26a5-41c0-8f53-ec9afa71d997", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
192025-03-20T17:15:02.210ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
202025-03-20T17:15:02.212ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
212025-03-20T17:15:02.212ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 538.706503ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
222025-03-20T17:15:02.240ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
232025-03-20T17:15:02.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
242025-03-20T17:15:02.244ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
252025-03-20T17:15:02.244ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
262025-03-20T17:15:02.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
272025-03-20T17:15:02.260ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
282025-03-20T17:15:02.260ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "8705d5f3-862d-4455-a642-e4b0062f777f", "content-length": "134", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
292025-03-20T17:15:02.260ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
302025-03-20T17:15:02.260ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
312025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
322025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
332025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
342025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
352025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
362025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
372025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
382025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
392025-03-20T17:15:02.261ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
402025-03-20T17:15:02.261ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
412025-03-20T17:15:02.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
422025-03-20T17:15:02.265ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
432025-03-20T17:15:02.265ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c64f15-0a27-49b1-8131-5a8c839a2c1f", "content-length": "2", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
442025-03-20T17:15:02.265ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
452025-03-20T17:15:02.265ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "6977aad5-d995-4818-a1a0-e8ec38bb51a7", "content-length": "52", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
462025-03-20T17:15:02.265ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
472025-03-20T17:15:02.265ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
482025-03-20T17:15:02.308ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
492025-03-20T17:15:02.338ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f290b9-662a-4c9b-b01c-e8d3e970582c", "content-length": "372", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
502025-03-20T17:15:02.338ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
512025-03-20T17:15:02.338ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "221ee37c-4020-45f3-826d-2c1e4bfeb198", "content-length": "691", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
522025-03-20T17:15:02.338ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
532025-03-20T17:15:02.338ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
542025-03-20T17:15:02.338ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
552025-03-20T17:15:02.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
562025-03-20T17:15:02.339ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 665.518276ms iteration = 23 status = {}
572025-03-20T17:15:02.398ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
582025-03-20T17:15:02.398ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
592025-03-20T17:15:02.399ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1dd62dd9-0d3e-428c-90ce-8c01cda1bd46", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
602025-03-20T17:15:02.399ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
612025-03-20T17:15:02.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
622025-03-20T17:15:02.402ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
632025-03-20T17:15:02.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
642025-03-20T17:15:02.406ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
652025-03-20T17:15:02.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
662025-03-20T17:15:02.410ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
672025-03-20T17:15:02.410ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
682025-03-20T17:15:02.411ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a9e9ed95-ebe6-4d92-a3b0-afd6d61f2cd6", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
692025-03-20T17:15:02.413ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 729.311223ms iteration = 23 status = {}
702025-03-20T17:15:02.519ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "24ba6676-af71-4a64-85c7-d4f1f142a868", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
712025-03-20T17:15:02.519ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
722025-03-20T17:15:02.519ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
732025-03-20T17:15:02.520ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef0e234a-dde8-4e76-8cc7-12c2ee577ec9", "content-length": "682", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
742025-03-20T17:15:02.520ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
752025-03-20T17:15:02.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
762025-03-20T17:15:02.526ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
772025-03-20T17:15:02.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
782025-03-20T17:15:02.531ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
792025-03-20T17:15:02.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
802025-03-20T17:15:02.534ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
812025-03-20T17:15:02.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
822025-03-20T17:15:02.543ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 869.122313ms iteration = 23 status = {}
832025-03-20T17:15:02.895ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
842025-03-20T17:15:02.895ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
852025-03-20T17:15:02.935ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
862025-03-20T17:15:02.935ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:39905
872025-03-20T17:15:02.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:39905 req_id = e21aabfc-d6e9-4bc8-a9ae-3bc717a309ef roles = RoleSet { roles: {} } uri = /metrics/producers
882025-03-20T17:15:02.936ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:39905 req_id = e21aabfc-d6e9-4bc8-a9ae-3bc717a309ef resource = Database result = Ok(()) uri = /metrics/producers
892025-03-20T17:15:02.990ZINFOnexus: assigned collector to new producer collector_id = 6125e0e7-3df0-4e3f-a39d-9936863e5edc file = nexus/src/app/oximeter.rs:76 producer_id = 4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
902025-03-20T17:15:02.990ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 54649 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:39905 req_id = e21aabfc-d6e9-4bc8-a9ae-3bc717a309ef response_code = 201 uri = /metrics/producers
912025-03-20T17:15:02.990ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "e21aabfc-d6e9-4bc8-a9ae-3bc717a309ef", "content-length": "41", "date": "Thu, 20 Mar 2025 17:15:02 GMT"} })
922025-03-20T17:15:02.992ZDEBGnexus (producer-registration-task): registered with nexus successfully
932025-03-20T17:15:02.992ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
942025-03-20T17:15:04.941ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 241 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 9aebf9a6-f98d-4822-841e-c6d58bd6d9aa response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
952025-03-20T17:15:11.673ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
962025-03-20T17:15:11.673ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
972025-03-20T17:15:11.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
982025-03-20T17:15:11.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
992025-03-20T17:15:11.678ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1002025-03-20T17:15:11.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1012025-03-20T17:15:11.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1022025-03-20T17:15:11.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1032025-03-20T17:15:11.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1042025-03-20T17:15:11.680ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
1052025-03-20T17:15:11.680ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1062025-03-20T17:15:11.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1072025-03-20T17:15:11.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1082025-03-20T17:15:11.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
1092025-03-20T17:15:11.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1102025-03-20T17:15:11.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1112025-03-20T17:15:11.692ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
1122025-03-20T17:15:11.692ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.962798ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
1132025-03-20T17:15:11.709ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1142025-03-20T17:15:11.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1152025-03-20T17:15:11.711ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.320556ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1162025-03-20T17:15:14.933ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 03303b1e-2f18-431c-9694-66d0f7091b08 roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
1172025-03-20T17:15:14.934ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 03303b1e-2f18-431c-9694-66d0f7091b08 resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
1182025-03-20T17:15:14.935ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2185 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 03303b1e-2f18-431c-9694-66d0f7091b08 response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
1192025-03-20T17:15:14.936ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 7003d0e1-7774-4737-b984-58295bf710f0 roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
1202025-03-20T17:15:14.936ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 7003d0e1-7774-4737-b984-58295bf710f0 resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
1212025-03-20T17:15:14.938ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2095 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 7003d0e1-7774-4737-b984-58295bf710f0 response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
1222025-03-20T17:15:14.941ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 0ab3f842-d4ea-45d7-ada0-6723f458d5b3 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
1232025-03-20T17:15:21.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
1242025-03-20T17:15:21.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1252025-03-20T17:15:21.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1262025-03-20T17:15:21.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1272025-03-20T17:15:21.678ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1282025-03-20T17:15:21.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1292025-03-20T17:15:21.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1302025-03-20T17:15:21.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1312025-03-20T17:15:21.680ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
1322025-03-20T17:15:21.680ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1332025-03-20T17:15:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1342025-03-20T17:15:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1352025-03-20T17:15:21.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1362025-03-20T17:15:21.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1372025-03-20T17:15:21.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1382025-03-20T17:15:21.684ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.599497ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1392025-03-20T17:15:21.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
1402025-03-20T17:15:21.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1412025-03-20T17:15:21.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1422025-03-20T17:15:21.691ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
1432025-03-20T17:15:21.691ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.982718ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
1442025-03-20T17:15:24.942ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 220ac3d2-ca3f-4a42-95ef-0d378e8e4480 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
1452025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
1462025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
1472025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
1482025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
1492025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
1502025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1512025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1522025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
1532025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
1542025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1552025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1562025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
1572025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
1582025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1592025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1602025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
1612025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1622025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
1632025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1642025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1652025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1662025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1672025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1682025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1692025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1702025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1712025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1722025-03-20T17:15:31.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1732025-03-20T17:15:31.677ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1742025-03-20T17:15:31.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2765, tv_nsec: 139053945 } })
1752025-03-20T17:15:31.677ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1762025-03-20T17:15:31.677ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
1772025-03-20T17:15:31.677ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.527747ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
1782025-03-20T17:15:31.678ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1792025-03-20T17:15:31.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2765, tv_nsec: 141182148 } })
1802025-03-20T17:15:31.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
1812025-03-20T17:15:31.680ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1822025-03-20T17:15:31.680ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1832025-03-20T17:15:31.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1842025-03-20T17:15:31.680ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
1852025-03-20T17:15:31.680ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
1862025-03-20T17:15:31.680ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1872025-03-20T17:15:31.680ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1882025-03-20T17:15:31.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1892025-03-20T17:15:31.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1902025-03-20T17:15:31.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1912025-03-20T17:15:31.681ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1922025-03-20T17:15:31.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1932025-03-20T17:15:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1942025-03-20T17:15:31.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1952025-03-20T17:15:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1962025-03-20T17:15:31.684ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1972025-03-20T17:15:31.684ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
1982025-03-20T17:15:31.684ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
1992025-03-20T17:15:31.684ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
2002025-03-20T17:15:31.684ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
2012025-03-20T17:15:31.684ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2022025-03-20T17:15:31.684ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2032025-03-20T17:15:31.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2042025-03-20T17:15:31.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2052025-03-20T17:15:31.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2062025-03-20T17:15:31.686ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2072025-03-20T17:15:31.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2082025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
2092025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
2102025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
2112025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2122025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
2132025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
2142025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
2152025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
2162025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
2172025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
2182025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
2192025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
2202025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2212025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
2222025-03-20T17:15:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
2232025-03-20T17:15:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
2242025-03-20T17:15:31.688ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2252025-03-20T17:15:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
2262025-03-20T17:15:31.688ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
2272025-03-20T17:15:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2282025-03-20T17:15:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
2292025-03-20T17:15:31.691ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2302025-03-20T17:15:31.691ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2312025-03-20T17:15:31.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2322025-03-20T17:15:31.692ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2765, tv_nsec: 153647998 } })
2332025-03-20T17:15:31.692ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
2342025-03-20T17:15:31.692ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2352025-03-20T17:15:31.693ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 5.79991ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
2362025-03-20T17:15:31.694ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2372025-03-20T17:15:31.694ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2382025-03-20T17:15:31.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2392025-03-20T17:15:31.695ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
2402025-03-20T17:15:31.695ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2412025-03-20T17:15:31.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2422025-03-20T17:15:31.697ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.294507ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
2432025-03-20T17:15:31.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2442025-03-20T17:15:31.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2452025-03-20T17:15:31.698ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2462025-03-20T17:15:31.699ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
2472025-03-20T17:15:31.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
2482025-03-20T17:15:31.702ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
2492025-03-20T17:15:31.702ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2502025-03-20T17:15:31.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2512025-03-20T17:15:31.716ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
2522025-03-20T17:15:31.716ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 42.13745ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
2532025-03-20T17:15:31.726ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2542025-03-20T17:15:31.728ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
2552025-03-20T17:15:31.728ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2562025-03-20T17:15:31.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2572025-03-20T17:15:31.731ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
2582025-03-20T17:15:31.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
2592025-03-20T17:15:31.731ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2602025-03-20T17:15:31.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2612025-03-20T17:15:31.734ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2622025-03-20T17:15:31.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2632025-03-20T17:15:31.735ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2642025-03-20T17:15:31.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2652025-03-20T17:15:31.736ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 49.682648ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2662025-03-20T17:15:31.737ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2672025-03-20T17:15:31.741ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 59be64ba-8ced-4cf9-bc90-5b69ad6e044a, lookup_type: ById(59be64ba-8ced-4cf9-bc90-5b69ad6e044a) } result = Ok(())
2682025-03-20T17:15:31.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2692025-03-20T17:15:31.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2702025-03-20T17:15:31.742ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
2712025-03-20T17:15:31.742ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2722025-03-20T17:15:31.742ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "32fa80e6-a4e0-4cfc-b789-4d71d49952b7", "content-length": "26", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
2732025-03-20T17:15:31.742ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2742025-03-20T17:15:31.743ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2752025-03-20T17:15:31.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2762025-03-20T17:15:31.743ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2772025-03-20T17:15:31.743ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
2782025-03-20T17:15:31.743ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 63.023578ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
2792025-03-20T17:15:31.744ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 69.762389ms iteration = 24 status = 0
2802025-03-20T17:15:31.744ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2812025-03-20T17:15:31.745ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2822025-03-20T17:15:31.745ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
2832025-03-20T17:15:31.745ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2842025-03-20T17:15:31.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2852025-03-20T17:15:31.745ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2862025-03-20T17:15:31.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2872025-03-20T17:15:31.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2882025-03-20T17:15:31.747ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
2892025-03-20T17:15:31.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2902025-03-20T17:15:31.747ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2912025-03-20T17:15:31.747ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 60.062987ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
2922025-03-20T17:15:31.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2932025-03-20T17:15:31.747ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f49900ad-2010-4b4b-8b9d-a62c4002156d", "content-length": "26", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
2942025-03-20T17:15:31.747ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2952025-03-20T17:15:31.749ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2962025-03-20T17:15:31.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2972025-03-20T17:15:31.749ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2982025-03-20T17:15:31.750ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2992025-03-20T17:15:31.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3002025-03-20T17:15:31.751ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 64.20114ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
3012025-03-20T17:15:31.774ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3022025-03-20T17:15:31.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3032025-03-20T17:15:31.778ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3042025-03-20T17:15:31.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
3052025-03-20T17:15:31.781ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3062025-03-20T17:15:31.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3072025-03-20T17:15:31.784ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
3082025-03-20T17:15:31.784ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
3092025-03-20T17:15:31.784ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
3102025-03-20T17:15:31.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
3112025-03-20T17:15:31.785ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 110.938347ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
3122025-03-20T17:15:31.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3132025-03-20T17:15:31.790ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3142025-03-20T17:15:31.790ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3152025-03-20T17:15:31.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3162025-03-20T17:15:31.791ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 110.863866ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3172025-03-20T17:15:31.823ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
3182025-03-20T17:15:31.823ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 148.976585ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
3192025-03-20T17:15:31.828ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
3202025-03-20T17:15:31.828ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
3212025-03-20T17:15:31.828ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 143.881241ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
3222025-03-20T17:15:31.834ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 159.951569ms iteration = 26 status = {"physical_disks_added":0}
3232025-03-20T17:15:31.872ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
3242025-03-20T17:15:31.872ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3252025-03-20T17:15:31.874ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3262025-03-20T17:15:31.874ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 zone_address = fd00:1122:3344:101::2
3272025-03-20T17:15:31.874ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3282025-03-20T17:15:31.875ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e3663b1-177e-471c-a0c0-7df1a9359643", "content-length": "26", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3292025-03-20T17:15:31.875ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3302025-03-20T17:15:31.876ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 201.856647ms iteration = 24 status = {}
3312025-03-20T17:15:31.877ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3322025-03-20T17:15:31.877ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3332025-03-20T17:15:31.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3342025-03-20T17:15:31.877ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
3352025-03-20T17:15:31.897ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3362025-03-20T17:15:31.897ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8c0d82c-7ad7-4531-8192-f70d4578015e", "content-length": "1", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3372025-03-20T17:15:31.899ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3382025-03-20T17:15:31.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3392025-03-20T17:15:31.901ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 227.306673ms iteration = 24 status = null
3402025-03-20T17:15:31.976ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b39d8a5a-6f47-455d-87f7-4c159cd907b3", "content-length": "372", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3412025-03-20T17:15:31.976ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3422025-03-20T17:15:31.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3432025-03-20T17:15:32.006ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
3442025-03-20T17:15:32.007ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c49ca13-61bb-49a4-bf69-3678a55b98d7", "content-length": "691", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3452025-03-20T17:15:32.007ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
3462025-03-20T17:15:32.007ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
3472025-03-20T17:15:32.007ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
3482025-03-20T17:15:32.009ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 334.996886ms iteration = 24 status = {}
3492025-03-20T17:15:32.025ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
3502025-03-20T17:15:32.025ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
3512025-03-20T17:15:32.026ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e211bda4-e1b0-41eb-b9b0-68ddae7573a0", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3522025-03-20T17:15:32.026ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3532025-03-20T17:15:32.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3542025-03-20T17:15:32.028ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3552025-03-20T17:15:32.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3562025-03-20T17:15:32.031ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3572025-03-20T17:15:32.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3582025-03-20T17:15:32.034ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
3592025-03-20T17:15:32.034ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3602025-03-20T17:15:32.035ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "674742ab-1d16-4f05-8428-8889cda0de0c", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3612025-03-20T17:15:32.037ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 352.198606ms iteration = 24 status = {}
3622025-03-20T17:15:32.073ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3632025-03-20T17:15:32.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3642025-03-20T17:15:32.076ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
3652025-03-20T17:15:32.076ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3662025-03-20T17:15:32.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3672025-03-20T17:15:32.087ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
3682025-03-20T17:15:32.088ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "83ccc10b-afb1-4e71-89ec-1156ed23876f", "content-length": "134", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3692025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
3702025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
3712025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
3722025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
3732025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3742025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3752025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3762025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
3772025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3782025-03-20T17:15:32.088ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
3792025-03-20T17:15:32.089ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
3802025-03-20T17:15:32.089ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3812025-03-20T17:15:32.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3822025-03-20T17:15:32.091ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
3832025-03-20T17:15:32.091ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d91363-61c3-4df4-9d2f-7eeeb3801291", "content-length": "2", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3842025-03-20T17:15:32.091ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
3852025-03-20T17:15:32.091ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef980903-e8f6-44fe-8a37-fbcf503ed317", "content-length": "52", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3862025-03-20T17:15:32.091ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
3872025-03-20T17:15:32.091ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
3882025-03-20T17:15:32.184ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0ceb480a-bfdb-4452-8331-530d68f657ac", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3892025-03-20T17:15:32.184ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
3902025-03-20T17:15:32.184ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
3912025-03-20T17:15:32.184ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "58b41956-4169-46b4-9c64-b4e30b83722d", "content-length": "682", "date": "Thu, 20 Mar 2025 17:15:31 GMT"} })
3922025-03-20T17:15:32.184ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3932025-03-20T17:15:32.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3942025-03-20T17:15:32.188ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3952025-03-20T17:15:32.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3962025-03-20T17:15:32.192ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3972025-03-20T17:15:32.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3982025-03-20T17:15:32.194ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3992025-03-20T17:15:32.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4002025-03-20T17:15:32.201ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 527.383636ms iteration = 24 status = {}
4012025-03-20T17:15:34.942ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 6a73818b-a5e8-49bc-a02e-6448a40dfa09 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
4022025-03-20T17:15:41.673ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
4032025-03-20T17:15:41.673ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4042025-03-20T17:15:41.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4052025-03-20T17:15:41.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4062025-03-20T17:15:41.678ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4072025-03-20T17:15:41.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4082025-03-20T17:15:41.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4092025-03-20T17:15:41.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4102025-03-20T17:15:41.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4112025-03-20T17:15:41.680ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
4122025-03-20T17:15:41.680ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4132025-03-20T17:15:41.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4142025-03-20T17:15:41.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4152025-03-20T17:15:41.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
4162025-03-20T17:15:41.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4172025-03-20T17:15:41.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4182025-03-20T17:15:41.693ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
4192025-03-20T17:15:41.693ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.535104ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
4202025-03-20T17:15:41.710ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4212025-03-20T17:15:41.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4222025-03-20T17:15:41.741ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.557243ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4232025-03-20T17:15:44.934ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 2a9662ed-f3a7-4a07-ad1a-90afde6a5eae roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
4242025-03-20T17:15:44.935ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 2a9662ed-f3a7-4a07-ad1a-90afde6a5eae resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
4252025-03-20T17:15:44.938ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3471 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 2a9662ed-f3a7-4a07-ad1a-90afde6a5eae response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
4262025-03-20T17:15:44.938ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 60f4560c-8cef-4a1d-995e-99275119f52b roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
4272025-03-20T17:15:44.939ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 60f4560c-8cef-4a1d-995e-99275119f52b resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
4282025-03-20T17:15:44.940ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2362 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:37487 req_id = 60f4560c-8cef-4a1d-995e-99275119f52b response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
4292025-03-20T17:15:44.942ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 220 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 46c0e36f-4bd6-4031-878c-d596629896ff response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
4302025-03-20T17:15:51.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
4312025-03-20T17:15:51.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4322025-03-20T17:15:51.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4332025-03-20T17:15:51.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4342025-03-20T17:15:51.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4352025-03-20T17:15:51.679ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4362025-03-20T17:15:51.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4372025-03-20T17:15:51.681ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
4382025-03-20T17:15:51.681ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4392025-03-20T17:15:51.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4402025-03-20T17:15:51.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4412025-03-20T17:15:51.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4422025-03-20T17:15:51.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4432025-03-20T17:15:51.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4442025-03-20T17:15:51.683ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4452025-03-20T17:15:51.685ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.85772ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4462025-03-20T17:15:51.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
4472025-03-20T17:15:51.686ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4482025-03-20T17:15:51.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4492025-03-20T17:15:51.694ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
4502025-03-20T17:15:51.694ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.334052ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
4512025-03-20T17:15:54.942ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = dab33c73-029d-49aa-8d9f-46d292edf33b response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
4522025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
4532025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4542025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
4552025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
4562025-03-20T17:16:01.674ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-20 17:06:01.674765784 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
4572025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
4582025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4592025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
4602025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
4612025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4622025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
4632025-03-20T17:16:01.674ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = 24f105be-07da-48d8-9788-12335c722319
4642025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 18.22\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"24f105be-07da-48d8-9788-12335c722319"}
4652025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
4662025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
4672025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
4682025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4692025-03-20T17:16:01.674ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
4702025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
4712025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4722025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
4732025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
4742025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
4752025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
4762025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
4772025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
4782025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4792025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
4802025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
4812025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4822025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4832025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
4842025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
4852025-03-20T17:16:01.675ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
4862025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4872025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
4882025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4892025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
4902025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4912025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4922025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
4932025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4942025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4952025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
4962025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4972025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4982025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4992025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5002025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5012025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5022025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5032025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5042025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5052025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
5062025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5072025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5082025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5092025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5102025-03-20T17:16:01.675ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("10288fe4-7a66-4d9a-b35e-7ff8e78c5a67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("73606bac-bd57-4023-ab78-1abd01649856.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("73606bac-bd57-4023-ab78-1abd01649856.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 2795, tv_nsec: 136783931 } })
5112025-03-20T17:16:01.675ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("92b5ab97-8367-4fba-a6f3-a2c4de635fc6.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a5373d80-a752-4210-beb8-2199d52453e7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bbaf3ede-f1ea-4ca0-855f-b3040ff17ee7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bbaf3ede-f1ea-4ca0-855f-b3040ff17ee7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 2795, tv_nsec: 136773981 } })
5122025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5132025-03-20T17:16:01.675ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5142025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5152025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5162025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
5172025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.460725ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
5182025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
5192025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.516666ms iteration = 13 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
5202025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5212025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5222025-03-20T17:16:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5232025-03-20T17:16:01.681ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
5242025-03-20T17:16:01.681ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
5252025-03-20T17:16:01.681ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5262025-03-20T17:16:01.681ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5272025-03-20T17:16:01.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5282025-03-20T17:16:01.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5292025-03-20T17:16:01.683ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2795, tv_nsec: 144659594 } })
5302025-03-20T17:16:01.683ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
5312025-03-20T17:16:01.684ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 10.047525ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
5322025-03-20T17:16:01.684ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
5332025-03-20T17:16:01.684ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
5342025-03-20T17:16:01.684ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5352025-03-20T17:16:01.684ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
5362025-03-20T17:16:01.684ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
5372025-03-20T17:16:01.684ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
5382025-03-20T17:16:01.684ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
5392025-03-20T17:16:01.685ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5402025-03-20T17:16:01.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5412025-03-20T17:16:01.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5422025-03-20T17:16:01.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
5432025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
5442025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
5452025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5462025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
5472025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
5482025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
5492025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
5502025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5512025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
5522025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
5532025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
5542025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
5552025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
5562025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
5572025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5582025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
5592025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
5602025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
5612025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
5622025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5632025-03-20T17:16:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
5642025-03-20T17:16:01.689ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5652025-03-20T17:16:01.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5662025-03-20T17:16:01.690ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5672025-03-20T17:16:01.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5682025-03-20T17:16:01.690ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-20 17:06:01.674765784 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
5692025-03-20T17:16:01.690ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 16.066528ms iteration = 13 status = {"expiration":"2025-03-20T17:06:01.674765784Z","pruned":[]}
5702025-03-20T17:16:01.690ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5712025-03-20T17:16:01.693ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2795, tv_nsec: 154227883 } })
5722025-03-20T17:16:01.693ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
5732025-03-20T17:16:01.695ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5742025-03-20T17:16:01.695ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5752025-03-20T17:16:01.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5762025-03-20T17:16:01.698ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5772025-03-20T17:16:01.701ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5782025-03-20T17:16:01.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5792025-03-20T17:16:01.701ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5802025-03-20T17:16:01.701ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5812025-03-20T17:16:01.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5822025-03-20T17:16:01.703ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5832025-03-20T17:16:01.705ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5842025-03-20T17:16:01.705ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5852025-03-20T17:16:01.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5862025-03-20T17:16:01.706ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5872025-03-20T17:16:01.706ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5882025-03-20T17:16:01.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5892025-03-20T17:16:01.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5902025-03-20T17:16:01.708ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5912025-03-20T17:16:01.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5922025-03-20T17:16:01.714ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5932025-03-20T17:16:01.714ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5942025-03-20T17:16:01.717ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
5952025-03-20T17:16:01.717ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5962025-03-20T17:16:01.717ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5972025-03-20T17:16:01.717ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5982025-03-20T17:16:01.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5992025-03-20T17:16:01.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6002025-03-20T17:16:01.720ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 33.398918ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
6012025-03-20T17:16:01.720ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 33.646941ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
6022025-03-20T17:16:01.720ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6032025-03-20T17:16:01.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6042025-03-20T17:16:01.723ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6052025-03-20T17:16:01.723ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6062025-03-20T17:16:01.723ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6072025-03-20T17:16:01.727ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
6082025-03-20T17:16:01.727ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
6092025-03-20T17:16:01.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
6102025-03-20T17:16:01.728ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6112025-03-20T17:16:01.729ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6122025-03-20T17:16:01.732ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6132025-03-20T17:16:01.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6142025-03-20T17:16:01.733ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6152025-03-20T17:16:01.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6162025-03-20T17:16:01.736ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2795, tv_nsec: 197544646 } })
6172025-03-20T17:16:01.736ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
6182025-03-20T17:16:01.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6192025-03-20T17:16:01.742ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6202025-03-20T17:16:01.742ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6212025-03-20T17:16:01.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6222025-03-20T17:16:01.745ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6232025-03-20T17:16:01.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6242025-03-20T17:16:01.752ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6252025-03-20T17:16:01.755ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
6262025-03-20T17:16:01.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
6272025-03-20T17:16:01.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
6282025-03-20T17:16:01.756ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6292025-03-20T17:16:01.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6302025-03-20T17:16:01.758ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6312025-03-20T17:16:01.759ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6322025-03-20T17:16:01.761ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6332025-03-20T17:16:01.762ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
6342025-03-20T17:16:01.762ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
6352025-03-20T17:16:01.763ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
6362025-03-20T17:16:01.763ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6372025-03-20T17:16:01.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
6382025-03-20T17:16:01.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6392025-03-20T17:16:01.764ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
6402025-03-20T17:16:01.764ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6412025-03-20T17:16:01.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6422025-03-20T17:16:01.769ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6432025-03-20T17:16:01.771ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6442025-03-20T17:16:01.773ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6452025-03-20T17:16:01.773ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6462025-03-20T17:16:01.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6472025-03-20T17:16:01.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
6482025-03-20T17:16:01.775ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
6492025-03-20T17:16:01.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
6502025-03-20T17:16:01.778ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
6512025-03-20T17:16:01.778ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
6522025-03-20T17:16:01.778ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
6532025-03-20T17:16:01.778ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6542025-03-20T17:16:01.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6552025-03-20T17:16:01.779ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 104.602641ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6562025-03-20T17:16:01.780ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 93.321094ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
6572025-03-20T17:16:01.781ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6582025-03-20T17:16:01.781ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
6592025-03-20T17:16:01.781ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6602025-03-20T17:16:01.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6612025-03-20T17:16:01.782ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6622025-03-20T17:16:01.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6632025-03-20T17:16:01.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6642025-03-20T17:16:01.783ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6652025-03-20T17:16:01.783ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6662025-03-20T17:16:01.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6672025-03-20T17:16:01.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6682025-03-20T17:16:01.786ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
6692025-03-20T17:16:01.786ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6702025-03-20T17:16:01.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6712025-03-20T17:16:01.787ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 100.702ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
6722025-03-20T17:16:01.788ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
6732025-03-20T17:16:01.788ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
6742025-03-20T17:16:01.788ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
6752025-03-20T17:16:01.788ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6762025-03-20T17:16:01.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6772025-03-20T17:16:01.789ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
6782025-03-20T17:16:01.789ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
6792025-03-20T17:16:01.789ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 102.300187ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
6802025-03-20T17:16:01.789ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6812025-03-20T17:16:01.790ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
6822025-03-20T17:16:01.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
6832025-03-20T17:16:01.791ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6842025-03-20T17:16:01.791ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6852025-03-20T17:16:01.792ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6862025-03-20T17:16:01.792ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6872025-03-20T17:16:01.792ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
6882025-03-20T17:16:01.792ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6892025-03-20T17:16:01.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6902025-03-20T17:16:01.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6912025-03-20T17:16:01.793ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
6922025-03-20T17:16:01.794ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 119.251514ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
6932025-03-20T17:16:01.794ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
6942025-03-20T17:16:01.794ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6952025-03-20T17:16:01.794ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
6962025-03-20T17:16:01.794ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
6972025-03-20T17:16:01.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6982025-03-20T17:16:01.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
6992025-03-20T17:16:01.796ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
7002025-03-20T17:16:01.796ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7012025-03-20T17:16:01.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7022025-03-20T17:16:01.801ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 126.349139ms iteration = 6 status = {"nsuccess":5}
7032025-03-20T17:16:01.810ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 123.533349ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
7042025-03-20T17:16:01.821ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7052025-03-20T17:16:01.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7062025-03-20T17:16:01.823ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
7072025-03-20T17:16:01.823ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 148.820222ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
7082025-03-20T17:16:01.826ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7092025-03-20T17:16:01.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7102025-03-20T17:16:01.826ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7112025-03-20T17:16:01.829ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
7122025-03-20T17:16:01.831ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
7132025-03-20T17:16:01.831ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7142025-03-20T17:16:01.832ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7152025-03-20T17:16:01.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7162025-03-20T17:16:01.834ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 152.495311ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7172025-03-20T17:16:01.835ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
7182025-03-20T17:16:01.835ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
7192025-03-20T17:16:01.835ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 159.93938ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
7202025-03-20T17:16:01.835ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
7212025-03-20T17:16:01.835ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7222025-03-20T17:16:01.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7232025-03-20T17:16:01.851ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 163.91502ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
7242025-03-20T17:16:01.851ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
7252025-03-20T17:16:01.851ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
7262025-03-20T17:16:01.851ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 166.820371ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7272025-03-20T17:16:01.864ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7282025-03-20T17:16:01.864ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7292025-03-20T17:16:01.881ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
7302025-03-20T17:16:01.881ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7312025-03-20T17:16:01.882ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "59497124-6685-47eb-93e5-f1eba316d1ed", "content-length": "26", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
7322025-03-20T17:16:01.882ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7332025-03-20T17:16:01.884ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7342025-03-20T17:16:01.932ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 256.985172ms iteration = 27 status = {"physical_disks_added":0}
7352025-03-20T17:16:01.996ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/319a3fc2-6412-49ab-b76f-7f14ef01a033/state
7362025-03-20T17:16:01.996ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/319a3fc2-6412-49ab-b76f-7f14ef01a033/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5660eca-d0e7-4052-98f6-26720c83bce4", "content-length": "130", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
7372025-03-20T17:16:02.006ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = f900baaf-5de7-409d-b181-48e8fc9136d2 sled_id = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-20T17:15:52.111829459Z }, migration_in: None, migration_out: None } vmm_id = 319a3fc2-6412-49ab-b76f-7f14ef01a033
7382025-03-20T17:16:02.006ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2088 instance_id = f900baaf-5de7-409d-b181-48e8fc9136d2 migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 319a3fc2-6412-49ab-b76f-7f14ef01a033 sled_id = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a vmm_id = 319a3fc2-6412-49ab-b76f-7f14ef01a033 vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-20T17:15:52.111829459Z }
7392025-03-20T17:16:02.006ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = f900baaf-5de7-409d-b181-48e8fc9136d2 roles = RoleSet { roles: {} } sled_id = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a vmm_id = 319a3fc2-6412-49ab-b76f-7f14ef01a033
7402025-03-20T17:16:02.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = f900baaf-5de7-409d-b181-48e8fc9136d2 resource = Database result = Ok(()) sled_id = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a vmm_id = 319a3fc2-6412-49ab-b76f-7f14ef01a033
7412025-03-20T17:16:02.009ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
7422025-03-20T17:16:02.039ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7432025-03-20T17:16:02.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7442025-03-20T17:16:02.049ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
7452025-03-20T17:16:02.050ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "734d1aa0-4313-47bd-9e16-383401c272ab", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7462025-03-20T17:16:02.050ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7472025-03-20T17:16:02.052ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "640b55e7-4667-4ddd-ab3e-20b9a2d33003", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7482025-03-20T17:16:02.052ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7492025-03-20T17:16:02.054ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7502025-03-20T17:16:02.054ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 379.42495ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
7512025-03-20T17:16:02.089ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
7522025-03-20T17:16:02.089ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7532025-03-20T17:16:02.089ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4783860-5d4c-48a8-a6a4-6a1f1e2ba34c", "content-length": "26", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
7542025-03-20T17:16:02.089ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7552025-03-20T17:16:02.091ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7562025-03-20T17:16:02.125ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 zone_address = fd00:1122:3344:101::2
7572025-03-20T17:16:02.125ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7582025-03-20T17:16:02.126ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1391dbd-01b0-4e79-874c-69db7783343c", "content-length": "26", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
7592025-03-20T17:16:02.126ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7602025-03-20T17:16:02.127ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
7612025-03-20T17:16:02.214ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
7622025-03-20T17:16:02.214ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7632025-03-20T17:16:02.217ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
7642025-03-20T17:16:02.217ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d44f50e8-2d0f-4b3e-9e56-9e88b49097cc dns_zone_name = oxide.test found_so_far = 0 version = 2
7652025-03-20T17:16:02.217ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7662025-03-20T17:16:02.217ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7672025-03-20T17:16:02.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7682025-03-20T17:16:02.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7692025-03-20T17:16:02.218ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5b0384e-6dda-4873-b40d-f6e8616a352f", "content-length": "585", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} })
7702025-03-20T17:16:02.218ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7712025-03-20T17:16:02.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7722025-03-20T17:16:02.219ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 544.733996ms iteration = 25 status = {}
7732025-03-20T17:16:02.219ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7742025-03-20T17:16:02.220ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7752025-03-20T17:16:02.221ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
7762025-03-20T17:16:02.222ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
7772025-03-20T17:16:02.222ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7782025-03-20T17:16:02.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7792025-03-20T17:16:02.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 59be64ba-8ced-4cf9-bc90-5b69ad6e044a, lookup_type: ById(59be64ba-8ced-4cf9-bc90-5b69ad6e044a) } result = Ok(())
7802025-03-20T17:16:02.223ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7812025-03-20T17:16:02.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7822025-03-20T17:16:02.225ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 550.87971ms iteration = 25 status = 0
7832025-03-20T17:16:02.239ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7842025-03-20T17:16:02.239ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "79b14b76-3f0b-4122-b73b-9bd8321bac35", "content-length": "1", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
7852025-03-20T17:16:02.240ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7862025-03-20T17:16:02.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7872025-03-20T17:16:02.242ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 568.10172ms iteration = 25 status = null
7882025-03-20T17:16:02.250ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7892025-03-20T17:16:02.253ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
7902025-03-20T17:16:02.253ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7912025-03-20T17:16:02.253ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
7922025-03-20T17:16:02.253ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 02f70c4b-db30-4840-b4d1-314d13a3510d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
7932025-03-20T17:16:02.253ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7942025-03-20T17:16:02.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7952025-03-20T17:16:02.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7962025-03-20T17:16:02.255ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d44f50e8-2d0f-4b3e-9e56-9e88b49097cc dns_zone_name = oxide.test found_so_far = 1 version = 2
7972025-03-20T17:16:02.255ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
7982025-03-20T17:16:02.255ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-20 17:04:53.696227 UTC dns_group = external generation = 2
7992025-03-20T17:16:02.255ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 580.905744ms iteration = 13 status = {"generation":2}
8002025-03-20T17:16:02.258ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-20T17:04:00.028646Z, time_modified: 2025-03-20T17:04:00.028646Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ca06964a-7ce0-46e9-890b-0b57019b1afc, db_silo: Silo { identity: SiloIdentity { id: ca06964a-7ce0-46e9-890b-0b57019b1afc, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-20T17:04:53.538118Z, time_modified: 2025-03-20T17:04:53.538118Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ca06964a-7ce0-46e9-890b-0b57019b1afc, db_silo: Silo { identity: SiloIdentity { id: ca06964a-7ce0-46e9-890b-0b57019b1afc, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-20T17:04:53.538118Z, time_modified: 2025-03-20T17:04:53.538118Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
8012025-03-20T17:16:02.258ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 583.560811ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ca06964a-7ce0-46e9-890b-0b57019b1afc","tls_certs":["450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63"]}},"default_endpoint":{"silo_id":"ca06964a-7ce0-46e9-890b-0b57019b1afc","tls_certs":["450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
8022025-03-20T17:16:02.262ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {784948f1-93b6-43e5-acd7-fca8be06c586: {10.151.1.176: {30d260e6-6109-4556-a7fd-f23e92320c83}, 10.151.1.175: {30d260e6-6109-4556-a7fd-f23e92320c83}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
8032025-03-20T17:16:02.262ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
8042025-03-20T17:16:02.263ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c630b9c1-30e7-43f9-a213-e39dc38e0abe", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} })
8052025-03-20T17:16:02.263ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8062025-03-20T17:16:02.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8072025-03-20T17:16:02.266ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8082025-03-20T17:16:02.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8092025-03-20T17:16:02.269ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8102025-03-20T17:16:02.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8112025-03-20T17:16:02.271ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8122025-03-20T17:16:02.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8132025-03-20T17:16:02.272ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8142025-03-20T17:16:02.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8152025-03-20T17:16:02.274ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8162025-03-20T17:16:02.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8172025-03-20T17:16:02.276ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8182025-03-20T17:16:02.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8192025-03-20T17:16:02.279ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
8202025-03-20T17:16:02.280ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
8212025-03-20T17:16:02.280ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
8222025-03-20T17:16:02.280ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
8232025-03-20T17:16:02.281ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 599.60297ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"running":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
8242025-03-20T17:16:02.281ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "9eb75d71-5a32-4dfb-a6bc-cd250dcf558a", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} })
8252025-03-20T17:16:02.282ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 598.020183ms iteration = 25 status = {}
8262025-03-20T17:16:02.285ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8272025-03-20T17:16:02.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8282025-03-20T17:16:02.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8292025-03-20T17:16:02.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8302025-03-20T17:16:02.291ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 02f70c4b-db30-4840-b4d1-314d13a3510d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
8312025-03-20T17:16:02.291ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
8322025-03-20T17:16:02.291ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-20 17:04:53.071389 UTC dns_group = internal generation = 1
8332025-03-20T17:16:02.291ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 617.048871ms iteration = 13 status = {"generation":1}
8342025-03-20T17:16:02.298ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
8352025-03-20T17:16:02.298ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1c4ded8f-1954-41b1-b539-b3a73eb9f0eb", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8362025-03-20T17:16:02.298ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8372025-03-20T17:16:02.300ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "8b04eaec-6cfd-454a-bc0c-d07ed3ab8b9b", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8382025-03-20T17:16:02.300ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8392025-03-20T17:16:02.302ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "493b1862-1988-4a83-9607-87f48bce1aa2", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8402025-03-20T17:16:02.302ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8412025-03-20T17:16:02.304ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8422025-03-20T17:16:02.304ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8432025-03-20T17:16:02.304ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 629.3182ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
8442025-03-20T17:16:02.305ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
8452025-03-20T17:16:02.305ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 621.027783ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
8462025-03-20T17:16:02.306ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
8472025-03-20T17:16:02.306ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8482025-03-20T17:16:02.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8492025-03-20T17:16:02.308ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8502025-03-20T17:16:02.310ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
8512025-03-20T17:16:02.310ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8522025-03-20T17:16:02.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8532025-03-20T17:16:02.411ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8542025-03-20T17:16:02.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8552025-03-20T17:16:02.413ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
8562025-03-20T17:16:02.413ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8572025-03-20T17:16:02.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8582025-03-20T17:16:02.417ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
8592025-03-20T17:16:02.417ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "b11ad2ae-d28e-4a73-91c6-fdb9d67e42d3", "content-length": "811", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} })
8602025-03-20T17:16:02.420ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(11365418) }} sled = buskin
8612025-03-20T17:16:02.420ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
8622025-03-20T17:16:02.420ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
8632025-03-20T17:16:02.422ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 746.950547ms iteration = 25 status = {}
8642025-03-20T17:16:02.427ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
8652025-03-20T17:16:02.428ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cfe91fb-4318-4c31-8b4a-97950fc05b9e", "content-length": "134", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
8662025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
8672025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
8682025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
8692025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
8702025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8712025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8722025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8732025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
8742025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8752025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
8762025-03-20T17:16:02.428ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
8772025-03-20T17:16:02.428ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8782025-03-20T17:16:02.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8792025-03-20T17:16:02.430ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
8802025-03-20T17:16:02.430ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e4d1e3e-ba64-4851-bc0d-edf4418214f9", "content-length": "2", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
8812025-03-20T17:16:02.430ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
8822025-03-20T17:16:02.430ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ee71fd0-17dd-404e-a801-66c4ec09cacc", "content-length": "52", "date": "Thu, 20 Mar 2025 17:16:01 GMT"} })
8832025-03-20T17:16:02.431ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
8842025-03-20T17:16:02.431ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
8852025-03-20T17:16:02.536ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b5856994-8ae4-4f14-8872-505ddd491414", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} })
8862025-03-20T17:16:02.536ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
8872025-03-20T17:16:02.536ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
8882025-03-20T17:16:02.536ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "625b7e9e-6078-44dc-86ea-227a0800891b", "content-length": "682", "date": "Thu, 20 Mar 2025 17:16:02 GMT"} })
8892025-03-20T17:16:02.536ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8902025-03-20T17:16:02.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8912025-03-20T17:16:02.540ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8922025-03-20T17:16:02.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8932025-03-20T17:16:02.543ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8942025-03-20T17:16:02.544ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8952025-03-20T17:16:02.546ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8962025-03-20T17:16:02.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8972025-03-20T17:16:02.553ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 878.684532ms iteration = 25 status = {}
8982025-03-20T17:16:04.945ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2697 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 351bf8e4-26ab-4b65-892f-0b579f2ced74 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
8992025-03-20T17:16:11.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
9002025-03-20T17:16:11.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9012025-03-20T17:16:11.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9022025-03-20T17:16:11.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9032025-03-20T17:16:11.678ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9042025-03-20T17:16:11.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9052025-03-20T17:16:11.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9062025-03-20T17:16:11.681ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
9072025-03-20T17:16:11.681ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9082025-03-20T17:16:11.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9092025-03-20T17:16:11.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9102025-03-20T17:16:11.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9112025-03-20T17:16:11.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9122025-03-20T17:16:11.683ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9132025-03-20T17:16:11.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9142025-03-20T17:16:11.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
9152025-03-20T17:16:11.686ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9162025-03-20T17:16:11.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9172025-03-20T17:16:11.695ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
9182025-03-20T17:16:11.695ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.942508ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
9192025-03-20T17:16:11.713ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.470969ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9202025-03-20T17:16:14.943ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 6589fe28-901f-4a7c-87d9-d8c1d4af8cf1 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
9212025-03-20T17:16:21.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
9222025-03-20T17:16:21.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9232025-03-20T17:16:21.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9242025-03-20T17:16:21.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9252025-03-20T17:16:21.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9262025-03-20T17:16:21.679ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9272025-03-20T17:16:21.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9282025-03-20T17:16:21.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9292025-03-20T17:16:21.681ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
9302025-03-20T17:16:21.681ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9312025-03-20T17:16:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9322025-03-20T17:16:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9332025-03-20T17:16:21.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9342025-03-20T17:16:21.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9352025-03-20T17:16:21.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9362025-03-20T17:16:21.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
9372025-03-20T17:16:21.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9382025-03-20T17:16:21.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9392025-03-20T17:16:21.693ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
9402025-03-20T17:16:21.693ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.627344ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
9412025-03-20T17:16:21.714ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.768582ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9422025-03-20T17:16:24.943ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 307 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 8c679a94-43ad-4041-8ca0-8f18ac7e9bf8 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
9432025-03-20T17:16:31.674ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
9442025-03-20T17:16:31.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
9452025-03-20T17:16:31.674ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9462025-03-20T17:16:31.674ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
9472025-03-20T17:16:31.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9482025-03-20T17:16:31.674ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
9492025-03-20T17:16:31.674ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
9502025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
9512025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9522025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9532025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
9542025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
9552025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
9562025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9572025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9582025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9592025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
9602025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9612025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
9622025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9632025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
9642025-03-20T17:16:31.675ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9652025-03-20T17:16:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9662025-03-20T17:16:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9672025-03-20T17:16:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9682025-03-20T17:16:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9692025-03-20T17:16:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9702025-03-20T17:16:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9712025-03-20T17:16:31.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2825, tv_nsec: 138388268 } })
9722025-03-20T17:16:31.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
9732025-03-20T17:16:31.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9742025-03-20T17:16:31.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2825, tv_nsec: 139417019 } })
9752025-03-20T17:16:31.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
9762025-03-20T17:16:31.679ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9772025-03-20T17:16:31.679ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.925041ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9782025-03-20T17:16:31.679ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9792025-03-20T17:16:31.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
9802025-03-20T17:16:31.682ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
9812025-03-20T17:16:31.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9822025-03-20T17:16:31.682ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9832025-03-20T17:16:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9842025-03-20T17:16:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9852025-03-20T17:16:31.683ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9862025-03-20T17:16:31.683ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9872025-03-20T17:16:31.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9882025-03-20T17:16:31.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9892025-03-20T17:16:31.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9902025-03-20T17:16:31.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9912025-03-20T17:16:31.684ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9922025-03-20T17:16:31.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9932025-03-20T17:16:31.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9942025-03-20T17:16:31.686ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
9952025-03-20T17:16:31.686ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
9962025-03-20T17:16:31.686ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
9972025-03-20T17:16:31.686ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9982025-03-20T17:16:31.686ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
9992025-03-20T17:16:31.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10002025-03-20T17:16:31.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10012025-03-20T17:16:31.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10022025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10032025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10042025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
10052025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
10062025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
10072025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
10082025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
10092025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
10102025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
10112025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10122025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
10132025-03-20T17:16:31.687ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
10142025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
10152025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
10162025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
10172025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10182025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10192025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
10202025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10212025-03-20T17:16:31.688ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10222025-03-20T17:16:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10232025-03-20T17:16:31.689ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10242025-03-20T17:16:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10252025-03-20T17:16:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10262025-03-20T17:16:31.689ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10272025-03-20T17:16:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10282025-03-20T17:16:31.690ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
10292025-03-20T17:16:31.690ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
10302025-03-20T17:16:31.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
10312025-03-20T17:16:31.693ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2825, tv_nsec: 153731159 } })
10322025-03-20T17:16:31.693ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
10332025-03-20T17:16:31.694ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 6.334137ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
10342025-03-20T17:16:31.697ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10352025-03-20T17:16:31.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10362025-03-20T17:16:31.721ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10372025-03-20T17:16:31.723ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10382025-03-20T17:16:31.723ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10392025-03-20T17:16:31.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10402025-03-20T17:16:31.728ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10412025-03-20T17:16:31.728ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10422025-03-20T17:16:31.728ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
10432025-03-20T17:16:31.728ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10442025-03-20T17:16:31.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10452025-03-20T17:16:31.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10462025-03-20T17:16:31.730ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 42.992458ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
10472025-03-20T17:16:31.731ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10482025-03-20T17:16:31.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10492025-03-20T17:16:31.733ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10502025-03-20T17:16:31.735ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
10512025-03-20T17:16:31.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
10522025-03-20T17:16:31.738ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 62.26241ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10532025-03-20T17:16:31.738ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 59be64ba-8ced-4cf9-bc90-5b69ad6e044a, lookup_type: ById(59be64ba-8ced-4cf9-bc90-5b69ad6e044a) } result = Ok(())
10542025-03-20T17:16:31.738ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10552025-03-20T17:16:31.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10562025-03-20T17:16:31.740ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 64.788826ms iteration = 26 status = 0
10572025-03-20T17:16:31.740ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10582025-03-20T17:16:31.743ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10592025-03-20T17:16:31.744ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.057938ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10602025-03-20T17:16:31.744ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10612025-03-20T17:16:31.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
10622025-03-20T17:16:31.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10632025-03-20T17:16:31.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10642025-03-20T17:16:31.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10652025-03-20T17:16:31.747ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10662025-03-20T17:16:31.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10672025-03-20T17:16:31.748ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10682025-03-20T17:16:31.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10692025-03-20T17:16:31.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10702025-03-20T17:16:31.748ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10712025-03-20T17:16:31.750ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10722025-03-20T17:16:31.750ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 68.703187ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
10732025-03-20T17:16:31.751ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10742025-03-20T17:16:31.751ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10752025-03-20T17:16:31.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10762025-03-20T17:16:31.753ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 65.327291ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
10772025-03-20T17:16:31.754ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
10782025-03-20T17:16:31.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
10792025-03-20T17:16:31.756ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10802025-03-20T17:16:31.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10812025-03-20T17:16:31.758ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 70.423935ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10822025-03-20T17:16:31.758ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
10832025-03-20T17:16:31.758ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 84.220568ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
10842025-03-20T17:16:31.768ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
10852025-03-20T17:16:31.768ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
10862025-03-20T17:16:31.768ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e74ed532-349d-497a-a077-437c8a896d4e", "content-length": "26", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
10872025-03-20T17:16:31.768ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
10882025-03-20T17:16:31.770ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
10892025-03-20T17:16:31.776ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10902025-03-20T17:16:31.776ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
10912025-03-20T17:16:31.776ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10922025-03-20T17:16:31.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10932025-03-20T17:16:31.777ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10942025-03-20T17:16:31.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10952025-03-20T17:16:31.797ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10962025-03-20T17:16:31.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10972025-03-20T17:16:31.825ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10982025-03-20T17:16:31.825ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10992025-03-20T17:16:31.825ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
11002025-03-20T17:16:31.825ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 139.655427ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
11012025-03-20T17:16:31.827ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11022025-03-20T17:16:31.827ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11032025-03-20T17:16:31.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11042025-03-20T17:16:31.830ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11052025-03-20T17:16:31.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11062025-03-20T17:16:31.836ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 149.105205ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11072025-03-20T17:16:31.863ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
11082025-03-20T17:16:31.863ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 187.865761ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11092025-03-20T17:16:31.870ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 194.647851ms iteration = 28 status = {"physical_disks_added":0}
11102025-03-20T17:16:31.889ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
11112025-03-20T17:16:31.889ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11122025-03-20T17:16:31.889ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "49621f0b-4e00-4d32-a925-6050d7a8c993", "content-length": "372", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11132025-03-20T17:16:31.889ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11142025-03-20T17:16:31.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11152025-03-20T17:16:31.904ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
11162025-03-20T17:16:31.904ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11172025-03-20T17:16:31.904ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "04863d10-f1b5-496b-9d99-a50c2a42fc16", "content-length": "26", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11182025-03-20T17:16:31.904ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11192025-03-20T17:16:31.906ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11202025-03-20T17:16:31.927ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11212025-03-20T17:16:31.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11222025-03-20T17:16:31.929ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11232025-03-20T17:16:31.931ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11242025-03-20T17:16:31.931ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11252025-03-20T17:16:31.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11262025-03-20T17:16:31.931ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
11272025-03-20T17:16:31.931ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
11282025-03-20T17:16:31.932ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "3c8c687d-0673-4765-9ade-afd8babc1806", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11292025-03-20T17:16:31.932ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11302025-03-20T17:16:31.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11312025-03-20T17:16:31.933ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11322025-03-20T17:16:31.933ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11332025-03-20T17:16:31.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11342025-03-20T17:16:31.934ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11352025-03-20T17:16:31.934ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11362025-03-20T17:16:31.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11372025-03-20T17:16:31.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11382025-03-20T17:16:31.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11392025-03-20T17:16:31.939ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
11402025-03-20T17:16:31.939ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11412025-03-20T17:16:31.940ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 zone_address = fd00:1122:3344:101::2
11422025-03-20T17:16:31.940ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11432025-03-20T17:16:31.940ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fdd8fcb-703d-49ae-a5b3-27e7e0804298", "content-length": "26", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11442025-03-20T17:16:31.940ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11452025-03-20T17:16:31.942ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
11462025-03-20T17:16:31.976ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11472025-03-20T17:16:31.977ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f53cab-e5c2-42d2-a4b7-9acfc5164270", "content-length": "2", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11482025-03-20T17:16:31.979ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11492025-03-20T17:16:31.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11502025-03-20T17:16:31.981ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 307.008844ms iteration = 26 status = null
11512025-03-20T17:16:31.996ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 321.334783ms iteration = 26 status = {}
11522025-03-20T17:16:32.041ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "bcc95f3b-e056-446b-9f05-aceafe122523", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11532025-03-20T17:16:32.043ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 356.961795ms iteration = 26 status = {}
11542025-03-20T17:16:32.059ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
11552025-03-20T17:16:32.059ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "991593be-f9c2-484f-abf5-a5a59e1f8aa0", "content-length": "691", "date": "Thu, 20 Mar 2025 17:16:32 GMT"} })
11562025-03-20T17:16:32.059ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
11572025-03-20T17:16:32.059ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
11582025-03-20T17:16:32.059ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
11592025-03-20T17:16:32.061ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 385.701466ms iteration = 26 status = {}
11602025-03-20T17:16:32.100ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11612025-03-20T17:16:32.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11622025-03-20T17:16:32.102ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
11632025-03-20T17:16:32.102ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11642025-03-20T17:16:32.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11652025-03-20T17:16:32.113ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
11662025-03-20T17:16:32.114ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "646233fb-065d-42ca-ad0f-fa0642a6187e", "content-length": "134", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11672025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
11682025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
11692025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
11702025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
11712025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11722025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11732025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11742025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
11752025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11762025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
11772025-03-20T17:16:32.114ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
11782025-03-20T17:16:32.114ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11792025-03-20T17:16:32.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11802025-03-20T17:16:32.117ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
11812025-03-20T17:16:32.117ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fa72e63-b38f-47d9-a40e-793f1089809d", "content-length": "2", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11822025-03-20T17:16:32.117ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
11832025-03-20T17:16:32.117ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8553dd0a-f483-4993-8e2f-3c9c1f42b41e", "content-length": "52", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11842025-03-20T17:16:32.117ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
11852025-03-20T17:16:32.117ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
11862025-03-20T17:16:32.219ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "cee80ee6-3da6-4f7c-87de-364875e380d6", "date": "Thu, 20 Mar 2025 17:16:32 GMT"} })
11872025-03-20T17:16:32.219ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
11882025-03-20T17:16:32.219ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
11892025-03-20T17:16:32.220ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "04cbaf69-c34b-4bda-ad8b-06368feb1f81", "content-length": "682", "date": "Thu, 20 Mar 2025 17:16:31 GMT"} })
11902025-03-20T17:16:32.220ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11912025-03-20T17:16:32.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11922025-03-20T17:16:32.223ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11932025-03-20T17:16:32.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11942025-03-20T17:16:32.226ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11952025-03-20T17:16:32.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11962025-03-20T17:16:32.228ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11972025-03-20T17:16:32.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11982025-03-20T17:16:32.235ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 559.711052ms iteration = 26 status = {}
11992025-03-20T17:16:34.943ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 274 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 1ff34f80-2b6a-4b3e-bd9a-3addce8731eb response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
12002025-03-20T17:16:41.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
12012025-03-20T17:16:41.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12022025-03-20T17:16:41.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12032025-03-20T17:16:41.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12042025-03-20T17:16:41.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12052025-03-20T17:16:41.679ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12062025-03-20T17:16:41.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12072025-03-20T17:16:41.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12082025-03-20T17:16:41.681ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
12092025-03-20T17:16:41.681ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12102025-03-20T17:16:41.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12112025-03-20T17:16:41.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12122025-03-20T17:16:41.683ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12132025-03-20T17:16:41.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
12142025-03-20T17:16:41.686ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12152025-03-20T17:16:41.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12162025-03-20T17:16:41.696ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
12172025-03-20T17:16:41.696ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.502994ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
12182025-03-20T17:16:41.712ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12192025-03-20T17:16:41.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12202025-03-20T17:16:41.713ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.830523ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12212025-03-20T17:16:44.943ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 265 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = c2ef6b86-ff3d-407c-a454-63824587dfa2 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
12222025-03-20T17:16:51.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
12232025-03-20T17:16:51.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12242025-03-20T17:16:51.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12252025-03-20T17:16:51.677ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12262025-03-20T17:16:51.680ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12272025-03-20T17:16:51.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12282025-03-20T17:16:51.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12292025-03-20T17:16:51.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12302025-03-20T17:16:51.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12312025-03-20T17:16:51.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
12322025-03-20T17:16:51.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12332025-03-20T17:16:51.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12342025-03-20T17:16:51.683ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12352025-03-20T17:16:51.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12362025-03-20T17:16:51.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12372025-03-20T17:16:51.685ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.233713ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12382025-03-20T17:16:51.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
12392025-03-20T17:16:51.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12402025-03-20T17:16:51.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12412025-03-20T17:16:51.693ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
12422025-03-20T17:16:51.693ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.955898ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
12432025-03-20T17:16:54.943ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 428 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 3ea18585-069f-41a1-ad72-fa5132b8a522 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
12442025-03-20T17:16:59.936ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:54729
12452025-03-20T17:16:59.936ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 8267be59-f12f-49dd-b65b-a02ca4e11de3 roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
12462025-03-20T17:16:59.936ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 8267be59-f12f-49dd-b65b-a02ca4e11de3 resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
12472025-03-20T17:16:59.940ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 4034 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 8267be59-f12f-49dd-b65b-a02ca4e11de3 response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
12482025-03-20T17:16:59.940ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 0fd5c463-f1de-4eac-bd75-37ef499c8e54 roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
12492025-03-20T17:16:59.940ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 0fd5c463-f1de-4eac-bd75-37ef499c8e54 resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
12502025-03-20T17:16:59.942ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2389 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 0fd5c463-f1de-4eac-bd75-37ef499c8e54 response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
12512025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
12522025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
12532025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
12542025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
12552025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12562025-03-20T17:17:01.675ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-20 17:07:01.675301479 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
12572025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
12582025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
12592025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12602025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
12612025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
12622025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
12632025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
12642025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
12652025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
12662025-03-20T17:17:01.675ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = 24f105be-07da-48d8-9788-12335c722319
12672025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
12682025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12692025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.36\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"24f105be-07da-48d8-9788-12335c722319"}
12702025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
12712025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
12722025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
12732025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
12742025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12752025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12762025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
12772025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12782025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12792025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
12802025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
12812025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12822025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
12832025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
12842025-03-20T17:17:01.675ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
12852025-03-20T17:17:01.675ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12862025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12872025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
12882025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
12892025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12902025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
12912025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12922025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
12932025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12942025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
12952025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12962025-03-20T17:17:01.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("10288fe4-7a66-4d9a-b35e-7ff8e78c5a67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("73606bac-bd57-4023-ab78-1abd01649856.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("73606bac-bd57-4023-ab78-1abd01649856.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 2855, tv_nsec: 136064564 } })
12972025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12982025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12992025-03-20T17:17:01.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("92b5ab97-8367-4fba-a6f3-a2c4de635fc6.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a5373d80-a752-4210-beb8-2199d52453e7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bbaf3ede-f1ea-4ca0-855f-b3040ff17ee7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bbaf3ede-f1ea-4ca0-855f-b3040ff17ee7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 2855, tv_nsec: 136126854 } })
13002025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
13012025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13022025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13032025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13042025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13052025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13062025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13072025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
13082025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
13092025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.383514ms iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
13102025-03-20T17:17:01.676ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.368334ms iteration = 14 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
13112025-03-20T17:17:01.678ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13122025-03-20T17:17:01.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13132025-03-20T17:17:01.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13142025-03-20T17:17:01.680ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13152025-03-20T17:17:01.680ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.76524ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
13162025-03-20T17:17:01.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13172025-03-20T17:17:01.680ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2855, tv_nsec: 140468190 } })
13182025-03-20T17:17:01.680ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-20 17:07:01.675301479 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
13192025-03-20T17:17:01.680ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.479647ms iteration = 14 status = {"expiration":"2025-03-20T17:07:01.675301479Z","pruned":[]}
13202025-03-20T17:17:01.680ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
13212025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13222025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13232025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
13242025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13252025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
13262025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13272025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13282025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13292025-03-20T17:17:01.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13302025-03-20T17:17:01.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13312025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
13322025-03-20T17:17:01.686ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
13332025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
13342025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
13352025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13362025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
13372025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
13382025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13392025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
13402025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13412025-03-20T17:17:01.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13422025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
13432025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
13442025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
13452025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
13462025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
13472025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
13482025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
13492025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13502025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
13512025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
13522025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
13532025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
13542025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
13552025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
13562025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
13572025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
13582025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13592025-03-20T17:17:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
13602025-03-20T17:17:01.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
13612025-03-20T17:17:01.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
13622025-03-20T17:17:01.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
13632025-03-20T17:17:01.690ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13642025-03-20T17:17:01.690ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13652025-03-20T17:17:01.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13662025-03-20T17:17:01.692ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2855, tv_nsec: 152248683 } })
13672025-03-20T17:17:01.692ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
13682025-03-20T17:17:01.693ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13692025-03-20T17:17:01.693ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13702025-03-20T17:17:01.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13712025-03-20T17:17:01.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
13722025-03-20T17:17:01.697ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
13732025-03-20T17:17:01.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
13742025-03-20T17:17:01.698ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
13752025-03-20T17:17:01.698ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13762025-03-20T17:17:01.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13772025-03-20T17:17:01.699ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
13782025-03-20T17:17:01.699ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13792025-03-20T17:17:01.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13802025-03-20T17:17:01.700ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
13812025-03-20T17:17:01.700ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13822025-03-20T17:17:01.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13832025-03-20T17:17:01.701ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13842025-03-20T17:17:01.701ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13852025-03-20T17:17:01.703ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13862025-03-20T17:17:01.703ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13872025-03-20T17:17:01.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13882025-03-20T17:17:01.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
13892025-03-20T17:17:01.705ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13902025-03-20T17:17:01.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13912025-03-20T17:17:01.706ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13922025-03-20T17:17:01.706ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13932025-03-20T17:17:01.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13942025-03-20T17:17:01.706ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13952025-03-20T17:17:01.707ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13962025-03-20T17:17:01.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13972025-03-20T17:17:01.709ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
13982025-03-20T17:17:01.709ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13992025-03-20T17:17:01.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
14002025-03-20T17:17:01.715ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14012025-03-20T17:17:01.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14022025-03-20T17:17:01.717ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 28.523518ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
14032025-03-20T17:17:01.718ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14042025-03-20T17:17:01.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14052025-03-20T17:17:01.719ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14062025-03-20T17:17:01.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14072025-03-20T17:17:01.721ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14082025-03-20T17:17:01.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14092025-03-20T17:17:01.722ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2855, tv_nsec: 182132315 } })
14102025-03-20T17:17:01.722ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
14112025-03-20T17:17:01.734ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
14122025-03-20T17:17:01.734ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 52.6934ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
14132025-03-20T17:17:01.764ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
14142025-03-20T17:17:01.764ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
14152025-03-20T17:17:01.764ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 88.430332ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
14162025-03-20T17:17:01.766ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
14172025-03-20T17:17:01.766ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
14182025-03-20T17:17:01.766ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
14192025-03-20T17:17:01.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14202025-03-20T17:17:01.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14212025-03-20T17:17:01.767ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14222025-03-20T17:17:01.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14232025-03-20T17:17:01.773ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14242025-03-20T17:17:01.773ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14252025-03-20T17:17:01.773ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
14262025-03-20T17:17:01.773ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14272025-03-20T17:17:01.773ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 85.472342ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
14282025-03-20T17:17:01.773ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14292025-03-20T17:17:01.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14302025-03-20T17:17:01.775ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14312025-03-20T17:17:01.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14322025-03-20T17:17:01.776ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 87.739495ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
14332025-03-20T17:17:01.777ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
14342025-03-20T17:17:01.777ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14352025-03-20T17:17:01.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
14362025-03-20T17:17:01.777ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14372025-03-20T17:17:01.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14382025-03-20T17:17:01.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14392025-03-20T17:17:01.779ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14402025-03-20T17:17:01.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
14412025-03-20T17:17:01.779ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14422025-03-20T17:17:01.779ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
14432025-03-20T17:17:01.779ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14442025-03-20T17:17:01.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14452025-03-20T17:17:01.781ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 92.748427ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
14462025-03-20T17:17:01.782ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14472025-03-20T17:17:01.782ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14482025-03-20T17:17:01.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14492025-03-20T17:17:01.783ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
14502025-03-20T17:17:01.783ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
14512025-03-20T17:17:01.783ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 94.480196ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
14522025-03-20T17:17:01.786ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14532025-03-20T17:17:01.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
14542025-03-20T17:17:01.787ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14552025-03-20T17:17:01.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14562025-03-20T17:17:01.787ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 99.098343ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
14572025-03-20T17:17:01.789ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
14582025-03-20T17:17:01.789ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14592025-03-20T17:17:01.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14602025-03-20T17:17:01.790ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 114.264492ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
14612025-03-20T17:17:01.790ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 108.026377ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14622025-03-20T17:17:01.790ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14632025-03-20T17:17:01.790ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14642025-03-20T17:17:01.794ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
14652025-03-20T17:17:01.794ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14662025-03-20T17:17:01.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14672025-03-20T17:17:01.796ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
14682025-03-20T17:17:01.796ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14692025-03-20T17:17:01.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14702025-03-20T17:17:01.811ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
14712025-03-20T17:17:01.811ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
14722025-03-20T17:17:01.811ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
14732025-03-20T17:17:01.811ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14742025-03-20T17:17:01.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14752025-03-20T17:17:01.812ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14762025-03-20T17:17:01.813ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14772025-03-20T17:17:01.813ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14782025-03-20T17:17:01.813ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14792025-03-20T17:17:01.816ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
14802025-03-20T17:17:01.816ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14812025-03-20T17:17:01.817ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14822025-03-20T17:17:01.818ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14832025-03-20T17:17:01.818ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14842025-03-20T17:17:01.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14852025-03-20T17:17:01.820ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
14862025-03-20T17:17:01.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
14872025-03-20T17:17:01.820ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
14882025-03-20T17:17:01.824ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14892025-03-20T17:17:01.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14902025-03-20T17:17:01.828ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14912025-03-20T17:17:01.828ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
14922025-03-20T17:17:01.828ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 02f70c4b-db30-4840-b4d1-314d13a3510d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
14932025-03-20T17:17:01.828ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14942025-03-20T17:17:01.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14952025-03-20T17:17:01.830ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14962025-03-20T17:17:01.832ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
14972025-03-20T17:17:01.832ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 157.480594ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
14982025-03-20T17:17:01.833ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
14992025-03-20T17:17:01.833ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 59be64ba-8ced-4cf9-bc90-5b69ad6e044a, lookup_type: ById(59be64ba-8ced-4cf9-bc90-5b69ad6e044a) } result = Ok(())
15002025-03-20T17:17:01.833ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15012025-03-20T17:17:01.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15022025-03-20T17:17:01.836ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
15032025-03-20T17:17:01.836ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15042025-03-20T17:17:01.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15052025-03-20T17:17:01.838ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 162.421145ms iteration = 27 status = 0
15062025-03-20T17:17:01.839ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 02f70c4b-db30-4840-b4d1-314d13a3510d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
15072025-03-20T17:17:01.839ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
15082025-03-20T17:17:01.839ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-20 17:04:53.071389 UTC dns_group = internal generation = 1
15092025-03-20T17:17:01.839ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 163.997921ms iteration = 14 status = {"generation":1}
15102025-03-20T17:17:01.841ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15112025-03-20T17:17:01.842ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
15122025-03-20T17:17:01.842ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d44f50e8-2d0f-4b3e-9e56-9e88b49097cc dns_zone_name = oxide.test found_so_far = 0 version = 2
15132025-03-20T17:17:01.842ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15142025-03-20T17:17:01.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15152025-03-20T17:17:01.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15162025-03-20T17:17:01.844ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15172025-03-20T17:17:01.844ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15182025-03-20T17:17:01.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15192025-03-20T17:17:01.847ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
15202025-03-20T17:17:01.847ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15212025-03-20T17:17:01.847ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15222025-03-20T17:17:01.847ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15232025-03-20T17:17:01.847ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
15242025-03-20T17:17:01.847ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 161.149782ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
15252025-03-20T17:17:01.849ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15262025-03-20T17:17:01.850ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d44f50e8-2d0f-4b3e-9e56-9e88b49097cc dns_zone_name = oxide.test found_so_far = 1 version = 2
15272025-03-20T17:17:01.850ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
15282025-03-20T17:17:01.850ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-20 17:04:53.696227 UTC dns_group = external generation = 2
15292025-03-20T17:17:01.850ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 175.054257ms iteration = 14 status = {"generation":2}
15302025-03-20T17:17:01.851ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
15312025-03-20T17:17:01.851ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 175.983346ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
15322025-03-20T17:17:01.852ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
15332025-03-20T17:17:01.852ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
15342025-03-20T17:17:01.852ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
15352025-03-20T17:17:01.880ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 192.180826ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
15362025-03-20T17:17:01.883ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 zone_address = fd00:1122:3344:101::2
15372025-03-20T17:17:01.883ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15382025-03-20T17:17:01.884ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a88d2b5-395c-4430-8947-6e555182ecb2", "content-length": "26", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
15392025-03-20T17:17:01.884ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15402025-03-20T17:17:01.886ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
15412025-03-20T17:17:01.893ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15422025-03-20T17:17:01.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15432025-03-20T17:17:01.895ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15442025-03-20T17:17:01.897ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-20T17:04:00.028646Z, time_modified: 2025-03-20T17:04:00.028646Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ca06964a-7ce0-46e9-890b-0b57019b1afc, db_silo: Silo { identity: SiloIdentity { id: ca06964a-7ce0-46e9-890b-0b57019b1afc, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-20T17:04:53.538118Z, time_modified: 2025-03-20T17:04:53.538118Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ca06964a-7ce0-46e9-890b-0b57019b1afc, db_silo: Silo { identity: SiloIdentity { id: ca06964a-7ce0-46e9-890b-0b57019b1afc, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-20T17:04:53.538118Z, time_modified: 2025-03-20T17:04:53.538118Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63" }] }) }
15452025-03-20T17:17:01.897ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 222.104848ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ca06964a-7ce0-46e9-890b-0b57019b1afc","tls_certs":["450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63"]}},"default_endpoint":{"silo_id":"ca06964a-7ce0-46e9-890b-0b57019b1afc","tls_certs":["450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
15462025-03-20T17:17:01.898ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
15472025-03-20T17:17:01.898ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15482025-03-20T17:17:01.898ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15492025-03-20T17:17:01.899ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 223.153979ms iteration = 29 status = {"physical_disks_added":0}
15502025-03-20T17:17:01.899ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15512025-03-20T17:17:01.901ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
15522025-03-20T17:17:01.901ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15532025-03-20T17:17:01.902ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15542025-03-20T17:17:01.915ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
15552025-03-20T17:17:01.915ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 229.236792ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
15562025-03-20T17:17:01.986ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
15572025-03-20T17:17:01.986ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15582025-03-20T17:17:01.986ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ec66489-3df6-461e-a113-ca8c959091eb", "content-length": "26", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
15592025-03-20T17:17:01.986ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15602025-03-20T17:17:01.988ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15612025-03-20T17:17:01.999ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
15622025-03-20T17:17:01.999ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15632025-03-20T17:17:01.999ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb1fc1c9-995d-4044-b951-7d8011e86426", "content-length": "26", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
15642025-03-20T17:17:01.999ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15652025-03-20T17:17:02.001ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15662025-03-20T17:17:02.014ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
15672025-03-20T17:17:02.014ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "ec1f1389-c40d-4a47-8723-4ef00335af07", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15682025-03-20T17:17:02.014ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15692025-03-20T17:17:02.016ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "66c2fcaa-a027-4f82-9df9-7415d0aea2bd", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15702025-03-20T17:17:02.016ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15712025-03-20T17:17:02.018ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15722025-03-20T17:17:02.018ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 342.763888ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
15732025-03-20T17:17:02.152ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
15742025-03-20T17:17:02.155ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
15752025-03-20T17:17:02.155ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d54655cf-b127-47f4-8472-23a6ae53aeb0", "content-length": "691", "date": "Thu, 20 Mar 2025 17:17:02 GMT"} })
15762025-03-20T17:17:02.155ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
15772025-03-20T17:17:02.155ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
15782025-03-20T17:17:02.155ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
15792025-03-20T17:17:02.157ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 481.038831ms iteration = 27 status = {}
15802025-03-20T17:17:02.175ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
15812025-03-20T17:17:02.175ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "358050c3-6482-471d-a965-1efcc829a379", "content-length": "2", "date": "Thu, 20 Mar 2025 17:17:02 GMT"} })
15822025-03-20T17:17:02.178ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15832025-03-20T17:17:02.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15842025-03-20T17:17:02.178ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 503.631747ms iteration = 27 status = {}
15852025-03-20T17:17:02.180ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 505.385435ms iteration = 27 status = null
15862025-03-20T17:17:02.270ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
15872025-03-20T17:17:02.270ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
15882025-03-20T17:17:02.270ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "311a66d2-ab78-4140-94d7-c38e2a1a968c", "content-length": "372", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
15892025-03-20T17:17:02.270ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15902025-03-20T17:17:02.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15912025-03-20T17:17:02.318ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
15922025-03-20T17:17:02.318ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
15932025-03-20T17:17:02.318ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "75acf426-78b5-4cfe-8963-323b173642d1", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
15942025-03-20T17:17:02.318ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15952025-03-20T17:17:02.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15962025-03-20T17:17:02.321ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15972025-03-20T17:17:02.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15982025-03-20T17:17:02.324ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15992025-03-20T17:17:02.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16002025-03-20T17:17:02.328ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
16012025-03-20T17:17:02.328ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
16022025-03-20T17:17:02.328ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "94cb13b7-f5fc-432f-95d3-1f476604ed00", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
16032025-03-20T17:17:02.330ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 643.886911ms iteration = 27 status = {}
16042025-03-20T17:17:02.355ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
16052025-03-20T17:17:02.356ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "caae87a9-2fdc-4fd4-aacc-3cd351ec4b5f", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
16062025-03-20T17:17:02.356ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
16072025-03-20T17:17:02.357ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "a1ab0786-b091-4f56-b1bb-f680b830c028", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
16082025-03-20T17:17:02.357ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
16092025-03-20T17:17:02.359ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "0e13f71d-1d7e-4c48-bf7f-406d0618ac28", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
16102025-03-20T17:17:02.359ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
16112025-03-20T17:17:02.361ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
16122025-03-20T17:17:02.361ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 686.409035ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
16132025-03-20T17:17:02.397ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16142025-03-20T17:17:02.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16152025-03-20T17:17:02.398ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
16162025-03-20T17:17:02.398ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16172025-03-20T17:17:02.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16182025-03-20T17:17:02.409ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
16192025-03-20T17:17:02.409ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c82d3bdc-0b88-48ee-b167-b8c793e0c381", "content-length": "134", "date": "Thu, 20 Mar 2025 17:17:02 GMT"} })
16202025-03-20T17:17:02.409ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
16212025-03-20T17:17:02.409ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
16222025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
16232025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
16242025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16252025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16262025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16272025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16282025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16292025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16302025-03-20T17:17:02.410ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
16312025-03-20T17:17:02.410ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16322025-03-20T17:17:02.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16332025-03-20T17:17:02.412ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
16342025-03-20T17:17:02.412ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "78134b05-bf73-4031-87f4-40b2ce3180b1", "content-length": "2", "date": "Thu, 20 Mar 2025 17:17:02 GMT"} })
16352025-03-20T17:17:02.412ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
16362025-03-20T17:17:02.412ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea99238b-c381-4540-be13-47d7e0e06e7f", "content-length": "52", "date": "Thu, 20 Mar 2025 17:17:02 GMT"} })
16372025-03-20T17:17:02.412ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
16382025-03-20T17:17:02.412ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
16392025-03-20T17:17:02.510ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "86ecdadf-874c-4c30-b9b4-f503e5d8bedb", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
16402025-03-20T17:17:02.510ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
16412025-03-20T17:17:02.510ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
16422025-03-20T17:17:02.510ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "45d9f9fe-caee-4792-9b3a-76908e90856a", "content-length": "682", "date": "Thu, 20 Mar 2025 17:17:01 GMT"} })
16432025-03-20T17:17:02.510ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16442025-03-20T17:17:02.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16452025-03-20T17:17:02.515ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16462025-03-20T17:17:02.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16472025-03-20T17:17:02.519ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16482025-03-20T17:17:02.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16492025-03-20T17:17:02.522ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16502025-03-20T17:17:02.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16512025-03-20T17:17:02.529ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 853.276117ms iteration = 27 status = {}
16522025-03-20T17:17:04.943ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 260 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 624270e9-a43e-4f4e-823d-a79011afeceb response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
16532025-03-20T17:17:11.675ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
16542025-03-20T17:17:11.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16552025-03-20T17:17:11.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16562025-03-20T17:17:11.677ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16572025-03-20T17:17:11.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16582025-03-20T17:17:11.679ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16592025-03-20T17:17:11.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16602025-03-20T17:17:11.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16612025-03-20T17:17:11.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16622025-03-20T17:17:11.683ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
16632025-03-20T17:17:11.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16642025-03-20T17:17:11.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16652025-03-20T17:17:11.684ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16662025-03-20T17:17:11.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16672025-03-20T17:17:11.685ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 2.84896ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16682025-03-20T17:17:11.710ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16692025-03-20T17:17:11.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
16702025-03-20T17:17:11.713ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16712025-03-20T17:17:11.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16722025-03-20T17:17:11.722ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
16732025-03-20T17:17:11.722ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 47.406195ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
16742025-03-20T17:17:14.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 395 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 600812e9-9e2e-4c6e-a67e-76ef81cd7042 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
16752025-03-20T17:17:21.676ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
16762025-03-20T17:17:21.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16772025-03-20T17:17:21.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16782025-03-20T17:17:21.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16792025-03-20T17:17:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16802025-03-20T17:17:21.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16812025-03-20T17:17:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16822025-03-20T17:17:21.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
16832025-03-20T17:17:21.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16842025-03-20T17:17:21.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16852025-03-20T17:17:21.683ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16862025-03-20T17:17:21.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16872025-03-20T17:17:21.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16882025-03-20T17:17:21.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16892025-03-20T17:17:21.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16902025-03-20T17:17:21.687ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
16912025-03-20T17:17:21.687ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16922025-03-20T17:17:21.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16932025-03-20T17:17:21.696ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
16942025-03-20T17:17:21.696ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.139831ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
16952025-03-20T17:17:21.714ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.051814ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16962025-03-20T17:17:24.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 498 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = bf1130f8-2b36-4d89-bf0e-b005d224331f response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
16972025-03-20T17:17:29.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2b507313-7a16-4632-89bc-b2b87e488c06 roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
16982025-03-20T17:17:29.935ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2b507313-7a16-4632-89bc-b2b87e488c06 resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
16992025-03-20T17:17:29.938ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3406 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2b507313-7a16-4632-89bc-b2b87e488c06 response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
17002025-03-20T17:17:29.938ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2fd863dc-6d4c-4877-8690-77a2a5e7e124 roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
17012025-03-20T17:17:29.939ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2fd863dc-6d4c-4877-8690-77a2a5e7e124 resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
17022025-03-20T17:17:29.941ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2469 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2fd863dc-6d4c-4877-8690-77a2a5e7e124 response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
17032025-03-20T17:17:31.674ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
17042025-03-20T17:17:31.674ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17052025-03-20T17:17:31.674ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
17062025-03-20T17:17:31.674ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
17072025-03-20T17:17:31.674ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17082025-03-20T17:17:31.674ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
17092025-03-20T17:17:31.674ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
17102025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17112025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
17122025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17132025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
17142025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
17152025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17162025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
17172025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
17182025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
17192025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17202025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
17212025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17222025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17232025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
17242025-03-20T17:17:31.675ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17252025-03-20T17:17:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17262025-03-20T17:17:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17272025-03-20T17:17:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17282025-03-20T17:17:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17292025-03-20T17:17:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17302025-03-20T17:17:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
17312025-03-20T17:17:31.677ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17322025-03-20T17:17:31.678ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17332025-03-20T17:17:31.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2885, tv_nsec: 139174106 } })
17342025-03-20T17:17:31.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2885, tv_nsec: 139217287 } })
17352025-03-20T17:17:31.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
17362025-03-20T17:17:31.679ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.616968ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
17372025-03-20T17:17:31.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
17382025-03-20T17:17:31.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
17392025-03-20T17:17:31.681ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17402025-03-20T17:17:31.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17412025-03-20T17:17:31.681ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
17422025-03-20T17:17:31.681ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17432025-03-20T17:17:31.681ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
17442025-03-20T17:17:31.681ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17452025-03-20T17:17:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17462025-03-20T17:17:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17472025-03-20T17:17:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17482025-03-20T17:17:31.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17492025-03-20T17:17:31.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17502025-03-20T17:17:31.685ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
17512025-03-20T17:17:31.685ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
17522025-03-20T17:17:31.685ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
17532025-03-20T17:17:31.685ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
17542025-03-20T17:17:31.685ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17552025-03-20T17:17:31.685ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17562025-03-20T17:17:31.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17572025-03-20T17:17:31.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17582025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17592025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
17602025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
17612025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
17622025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
17632025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17642025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
17652025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
17662025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
17672025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
17682025-03-20T17:17:31.687ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
17692025-03-20T17:17:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17702025-03-20T17:17:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
17712025-03-20T17:17:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
17722025-03-20T17:17:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
17732025-03-20T17:17:31.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
17742025-03-20T17:17:31.689ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17752025-03-20T17:17:31.690ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17762025-03-20T17:17:31.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17772025-03-20T17:17:31.690ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
17782025-03-20T17:17:31.690ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17792025-03-20T17:17:31.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17802025-03-20T17:17:31.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
17812025-03-20T17:17:31.691ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17822025-03-20T17:17:31.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17832025-03-20T17:17:31.697ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17842025-03-20T17:17:31.698ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17852025-03-20T17:17:31.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17862025-03-20T17:17:31.698ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17872025-03-20T17:17:31.698ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
17882025-03-20T17:17:31.698ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17892025-03-20T17:17:31.698ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17902025-03-20T17:17:31.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17912025-03-20T17:17:31.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17922025-03-20T17:17:31.700ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
17932025-03-20T17:17:31.700ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17942025-03-20T17:17:31.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17952025-03-20T17:17:31.705ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17962025-03-20T17:17:31.706ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
17972025-03-20T17:17:31.706ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17982025-03-20T17:17:31.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
17992025-03-20T17:17:31.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18002025-03-20T17:17:31.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18012025-03-20T17:17:31.707ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18022025-03-20T17:17:31.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18032025-03-20T17:17:31.710ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18042025-03-20T17:17:31.710ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
18052025-03-20T17:17:31.710ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18062025-03-20T17:17:31.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18072025-03-20T17:17:31.711ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 23.603707ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
18082025-03-20T17:17:31.725ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
18092025-03-20T17:17:31.725ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18102025-03-20T17:17:31.725ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18112025-03-20T17:17:31.726ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e8eac46-43a7-4248-85c3-dbfc1a6a92f0", "content-length": "26", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
18122025-03-20T17:17:31.726ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18132025-03-20T17:17:31.728ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18142025-03-20T17:17:31.728ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
18152025-03-20T17:17:31.728ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18162025-03-20T17:17:31.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18172025-03-20T17:17:31.730ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18182025-03-20T17:17:31.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18192025-03-20T17:17:31.732ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
18202025-03-20T17:17:31.732ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18212025-03-20T17:17:31.732ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18222025-03-20T17:17:31.732ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "25cb92a8-01cb-4db2-af3b-d471560d684f", "content-length": "26", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
18232025-03-20T17:17:31.733ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18242025-03-20T17:17:31.733ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18252025-03-20T17:17:31.733ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18262025-03-20T17:17:31.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18272025-03-20T17:17:31.735ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18282025-03-20T17:17:31.736ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
18292025-03-20T17:17:31.736ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18302025-03-20T17:17:31.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18312025-03-20T17:17:31.737ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
18322025-03-20T17:17:31.737ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18332025-03-20T17:17:31.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18342025-03-20T17:17:31.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18352025-03-20T17:17:31.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18362025-03-20T17:17:31.742ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2885, tv_nsec: 202851341 } })
18372025-03-20T17:17:31.742ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
18382025-03-20T17:17:31.744ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18392025-03-20T17:17:31.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18402025-03-20T17:17:31.747ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 59.275748ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
18412025-03-20T17:17:31.749ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.590633ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
18422025-03-20T17:17:31.749ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18432025-03-20T17:17:31.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18442025-03-20T17:17:31.750ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 62.704654ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
18452025-03-20T17:17:31.752ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18462025-03-20T17:17:31.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18472025-03-20T17:17:31.754ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18482025-03-20T17:17:31.755ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18492025-03-20T17:17:31.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18502025-03-20T17:17:31.756ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
18512025-03-20T17:17:31.756ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 74.403186ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
18522025-03-20T17:17:31.759ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
18532025-03-20T17:17:31.759ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18542025-03-20T17:17:31.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18552025-03-20T17:17:31.763ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18562025-03-20T17:17:31.765ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18572025-03-20T17:17:31.765ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18582025-03-20T17:17:31.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18592025-03-20T17:17:31.769ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18602025-03-20T17:17:31.769ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
18612025-03-20T17:17:31.769ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 84.017617ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
18622025-03-20T17:17:31.779ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
18632025-03-20T17:17:31.780ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 105.220018ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
18642025-03-20T17:17:31.782ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 100.574079ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18652025-03-20T17:17:31.786ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
18662025-03-20T17:17:31.786ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 110.16551ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
18672025-03-20T17:17:31.789ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18682025-03-20T17:17:31.792ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 59be64ba-8ced-4cf9-bc90-5b69ad6e044a, lookup_type: ById(59be64ba-8ced-4cf9-bc90-5b69ad6e044a) } result = Ok(())
18692025-03-20T17:17:31.792ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18702025-03-20T17:17:31.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18712025-03-20T17:17:31.794ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 118.917171ms iteration = 28 status = 0
18722025-03-20T17:17:31.799ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 123.910233ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
18732025-03-20T17:17:31.799ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 112.192751ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
18742025-03-20T17:17:31.803ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
18752025-03-20T17:17:31.804ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3ce2cb1-9e2e-43c6-89ff-0d6f96fb337f", "content-length": "2", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
18762025-03-20T17:17:31.806ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18772025-03-20T17:17:31.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
18782025-03-20T17:17:31.808ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 134.083209ms iteration = 28 status = null
18792025-03-20T17:17:31.816ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 142.175084ms iteration = 28 status = {}
18802025-03-20T17:17:31.824ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 zone_address = fd00:1122:3344:101::2
18812025-03-20T17:17:31.824ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18822025-03-20T17:17:31.824ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "adc345e2-d05d-4b61-af99-29e07e64e521", "content-length": "26", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
18832025-03-20T17:17:31.824ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18842025-03-20T17:17:31.827ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
18852025-03-20T17:17:31.836ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 160.631076ms iteration = 30 status = {"physical_disks_added":0}
18862025-03-20T17:17:31.857ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18872025-03-20T17:17:31.857ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18882025-03-20T17:17:31.902ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
18892025-03-20T17:17:31.902ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
18902025-03-20T17:17:31.904ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18912025-03-20T17:17:31.906ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
18922025-03-20T17:17:31.906ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18932025-03-20T17:17:31.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18942025-03-20T17:17:31.936ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18952025-03-20T17:17:31.938ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
18962025-03-20T17:17:31.938ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18972025-03-20T17:17:31.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18982025-03-20T17:17:31.941ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c47fddea-3ad4-4e83-b6b7-9c2ede767b39", "content-length": "372", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
18992025-03-20T17:17:31.941ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19002025-03-20T17:17:31.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19012025-03-20T17:17:31.951ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19022025-03-20T17:17:31.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19032025-03-20T17:17:31.954ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
19042025-03-20T17:17:31.954ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19052025-03-20T17:17:31.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19062025-03-20T17:17:31.964ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
19072025-03-20T17:17:31.964ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "34ba67c1-a882-4aa2-b0ff-2797c71581a2", "content-length": "134", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19082025-03-20T17:17:31.964ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
19092025-03-20T17:17:31.964ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
19102025-03-20T17:17:31.964ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
19112025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
19122025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19132025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19142025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19152025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
19162025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19172025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
19182025-03-20T17:17:31.965ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
19192025-03-20T17:17:31.965ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19202025-03-20T17:17:31.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19212025-03-20T17:17:31.967ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
19222025-03-20T17:17:31.968ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba59acf5-cd1c-4358-9760-e7a438638cb4", "content-length": "2", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19232025-03-20T17:17:31.968ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
19242025-03-20T17:17:31.968ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "edd71096-2054-4a65-8496-6aec522b6a04", "content-length": "52", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19252025-03-20T17:17:31.968ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
19262025-03-20T17:17:31.968ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
19272025-03-20T17:17:32.000ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
19282025-03-20T17:17:32.000ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
19292025-03-20T17:17:32.001ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e3670bfd-43d0-411c-8963-d868470b7661", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19302025-03-20T17:17:32.001ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19312025-03-20T17:17:32.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19322025-03-20T17:17:32.004ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19332025-03-20T17:17:32.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19342025-03-20T17:17:32.033ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
19352025-03-20T17:17:32.034ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "628b46bc-caca-4aae-855c-fc2abb7a7fc1", "content-length": "691", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19362025-03-20T17:17:32.034ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
19372025-03-20T17:17:32.034ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
19382025-03-20T17:17:32.034ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
19392025-03-20T17:17:32.034ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 358.853546ms iteration = 28 status = {}
19402025-03-20T17:17:32.036ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19412025-03-20T17:17:32.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19422025-03-20T17:17:32.040ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
19432025-03-20T17:17:32.040ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
19442025-03-20T17:17:32.044ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c6f2dd58-7743-4356-8cb6-4c68b6653a59", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19452025-03-20T17:17:32.045ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 359.879397ms iteration = 28 status = {}
19462025-03-20T17:17:32.143ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "89fdb5e7-c619-4107-b13f-e36b4b2bc2ea", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19472025-03-20T17:17:32.143ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
19482025-03-20T17:17:32.143ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
19492025-03-20T17:17:32.143ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "224de49e-2c8e-4e34-a54a-a2dacc5e8c13", "content-length": "682", "date": "Thu, 20 Mar 2025 17:17:31 GMT"} })
19502025-03-20T17:17:32.143ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19512025-03-20T17:17:32.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19522025-03-20T17:17:32.147ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19532025-03-20T17:17:32.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19542025-03-20T17:17:32.151ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19552025-03-20T17:17:32.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19562025-03-20T17:17:32.153ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19572025-03-20T17:17:32.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19582025-03-20T17:17:32.160ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 484.917052ms iteration = 28 status = {}
19592025-03-20T17:17:32.995ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
19602025-03-20T17:17:32.995ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
19612025-03-20T17:17:33.025ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
19622025-03-20T17:17:33.025ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:60193
19632025-03-20T17:17:33.026ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:60193 req_id = 21008c4d-6a81-4c9c-9a94-92cfcd71d6d3 roles = RoleSet { roles: {} } uri = /metrics/producers
19642025-03-20T17:17:33.026ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:60193 req_id = 21008c4d-6a81-4c9c-9a94-92cfcd71d6d3 resource = Database result = Ok(()) uri = /metrics/producers
19652025-03-20T17:17:33.063ZINFOnexus: assigned collector to new producer collector_id = 6125e0e7-3df0-4e3f-a39d-9936863e5edc file = nexus/src/app/oximeter.rs:76 producer_id = 4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
19662025-03-20T17:17:33.063ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 37335 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:60193 req_id = 21008c4d-6a81-4c9c-9a94-92cfcd71d6d3 response_code = 201 uri = /metrics/producers
19672025-03-20T17:17:33.063ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "21008c4d-6a81-4c9c-9a94-92cfcd71d6d3", "content-length": "41", "date": "Thu, 20 Mar 2025 17:17:33 GMT"} })
19682025-03-20T17:17:33.065ZDEBGnexus (producer-registration-task): registered with nexus successfully
19692025-03-20T17:17:33.065ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
19702025-03-20T17:17:34.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 323 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 1e03bc22-1efc-42fe-9289-ae99d5cc0162 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
19712025-03-20T17:17:41.675ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 83 reason = Timeout
19722025-03-20T17:17:41.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19732025-03-20T17:17:41.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19742025-03-20T17:17:41.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 83 reason = Timeout
19752025-03-20T17:17:41.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19762025-03-20T17:17:41.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19772025-03-20T17:17:41.706ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19782025-03-20T17:17:41.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19792025-03-20T17:17:41.708ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19802025-03-20T17:17:41.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19812025-03-20T17:17:41.710ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19822025-03-20T17:17:41.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19832025-03-20T17:17:41.711ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19842025-03-20T17:17:41.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19852025-03-20T17:17:41.712ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19862025-03-20T17:17:41.715ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
19872025-03-20T17:17:41.715ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19882025-03-20T17:17:41.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19892025-03-20T17:17:41.725ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
19902025-03-20T17:17:41.725ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 50.307305ms iteration = 83 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
19912025-03-20T17:17:41.738ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 56.180177ms iteration = 83 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19922025-03-20T17:17:44.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = ffe344b5-1432-4c24-a211-234a365cde38 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
19932025-03-20T17:17:51.675ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 84 reason = Timeout
19942025-03-20T17:17:51.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19952025-03-20T17:17:51.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19962025-03-20T17:17:51.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 84 reason = Timeout
19972025-03-20T17:17:51.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19982025-03-20T17:17:51.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19992025-03-20T17:17:51.704ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20002025-03-20T17:17:51.706ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20012025-03-20T17:17:51.706ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20022025-03-20T17:17:51.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20032025-03-20T17:17:51.709ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20042025-03-20T17:17:51.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20052025-03-20T17:17:51.711ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20062025-03-20T17:17:51.713ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20072025-03-20T17:17:51.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20082025-03-20T17:17:51.714ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
20092025-03-20T17:17:51.714ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20102025-03-20T17:17:51.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20112025-03-20T17:17:51.722ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
20122025-03-20T17:17:51.722ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 47.276994ms iteration = 84 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
20132025-03-20T17:17:51.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.021477ms iteration = 84 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20142025-03-20T17:17:54.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 287 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 66c5d393-ccd1-479f-b959-2f1840e57972 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
20152025-03-20T17:17:59.936ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2ae1d1f2-47f4-47f8-b97f-de41cea7e961 roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
20162025-03-20T17:17:59.936ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2ae1d1f2-47f4-47f8-b97f-de41cea7e961 resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
20172025-03-20T17:17:59.939ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3601 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2ae1d1f2-47f4-47f8-b97f-de41cea7e961 response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
20182025-03-20T17:17:59.940ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2322e3fc-08fd-495d-9ff3-aab9d472ec2a roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
20192025-03-20T17:17:59.940ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2322e3fc-08fd-495d-9ff3-aab9d472ec2a resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
20202025-03-20T17:17:59.942ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2599 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:54729 req_id = 2322e3fc-08fd-495d-9ff3-aab9d472ec2a response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
20212025-03-20T17:18:01.675ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 29 reason = Timeout
20222025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
20232025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 85 reason = Timeout
20242025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 16 reason = Timeout
20252025-03-20T17:18:01.676ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:82 reason = blueprint disabled target_id = 24f105be-07da-48d8-9788-12335c722319
20262025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20272025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.16\u{b5}s iteration = 16 status = {"enabled":false,"target_id":"24f105be-07da-48d8-9788-12335c722319"}
20282025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 15 reason = Timeout
20292025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 17 reason = Timeout
20302025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
20312025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20322025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 15 reason = Timeout
20332025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 15 reason = Timeout
20342025-03-20T17:18:01.676ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-20 17:08:01.676053364 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
20352025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
20362025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 15 reason = Timeout
20372025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20382025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 29 reason = Timeout
20392025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 15 reason = Timeout
20402025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 15 reason = Timeout
20412025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 29 reason = Timeout
20422025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
20432025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20442025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20452025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 17 reason = Timeout
20462025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 29 reason = Timeout
20472025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
20482025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 29 reason = Timeout
20492025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20502025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 29 reason = Timeout
20512025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20522025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 31 reason = Timeout
20532025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 15 reason = Timeout
20542025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20552025-03-20T17:18:01.676ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
20562025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 29 reason = Timeout
20572025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
20582025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20592025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 29 reason = Timeout
20602025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20612025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
20622025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20632025-03-20T17:18:01.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("10288fe4-7a66-4d9a-b35e-7ff8e78c5a67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("73606bac-bd57-4023-ab78-1abd01649856.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("73606bac-bd57-4023-ab78-1abd01649856.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 2915, tv_nsec: 136385297 } })
20642025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20652025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20662025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
20672025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
20682025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
20692025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20702025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20712025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20722025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20732025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
20742025-03-20T17:18:01.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("92b5ab97-8367-4fba-a6f3-a2c4de635fc6.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a5373d80-a752-4210-beb8-2199d52453e7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bbaf3ede-f1ea-4ca0-855f-b3040ff17ee7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bbaf3ede-f1ea-4ca0-855f-b3040ff17ee7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 2915, tv_nsec: 136588249 } })
20752025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20762025-03-20T17:18:01.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20772025-03-20T17:18:01.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
20782025-03-20T17:18:01.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.132462ms iteration = 15 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
20792025-03-20T17:18:01.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
20802025-03-20T17:18:01.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.331264ms iteration = 15 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
20812025-03-20T17:18:01.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20822025-03-20T17:18:01.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20832025-03-20T17:18:01.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2915, tv_nsec: 138954464 } })
20842025-03-20T17:18:01.679ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20852025-03-20T17:18:01.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
20862025-03-20T17:18:01.681ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20872025-03-20T17:18:01.682ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-20 17:08:01.676053364 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
20882025-03-20T17:18:01.682ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.969202ms iteration = 15 status = {"expiration":"2025-03-20T17:08:01.676053364Z","pruned":[]}
20892025-03-20T17:18:01.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 85 reason = Timeout
20902025-03-20T17:18:01.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20912025-03-20T17:18:01.682ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 29 reason = Timeout
20922025-03-20T17:18:01.682ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
20932025-03-20T17:18:01.682ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.63804ms iteration = 29 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
20942025-03-20T17:18:01.683ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2915, tv_nsec: 142755984 } })
20952025-03-20T17:18:01.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20962025-03-20T17:18:01.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
20972025-03-20T17:18:01.683ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
20982025-03-20T17:18:01.684ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20992025-03-20T17:18:01.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21002025-03-20T17:18:01.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21012025-03-20T17:18:01.684ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21022025-03-20T17:18:01.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21032025-03-20T17:18:01.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21042025-03-20T17:18:01.685ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21052025-03-20T17:18:01.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21062025-03-20T17:18:01.686ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 15 reason = Timeout
21072025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 29 reason = Timeout
21082025-03-20T17:18:01.687ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
21092025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 29 reason = Timeout
21102025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
21112025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21122025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
21132025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21142025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21152025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
21162025-03-20T17:18:01.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
21172025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 29 reason = Timeout
21182025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
21192025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 15 reason = Timeout
21202025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
21212025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 29 reason = Timeout
21222025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
21232025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 29 reason = Timeout
21242025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
21252025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 15 reason = Timeout
21262025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
21272025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 29 reason = Timeout
21282025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
21292025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 29 reason = Timeout
21302025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21312025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
21322025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
21332025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
21342025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
21352025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
21362025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21372025-03-20T17:18:01.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
21382025-03-20T17:18:01.694ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21392025-03-20T17:18:01.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
21402025-03-20T17:18:01.697ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
21412025-03-20T17:18:01.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
21422025-03-20T17:18:01.700ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21432025-03-20T17:18:01.702ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21442025-03-20T17:18:01.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21452025-03-20T17:18:01.703ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21462025-03-20T17:18:01.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
21472025-03-20T17:18:01.704ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
21482025-03-20T17:18:01.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
21492025-03-20T17:18:01.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21502025-03-20T17:18:01.715ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 26.711459ms iteration = 15 status = {"errors":[],"found_port_ok":[]}
21512025-03-20T17:18:01.749ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21522025-03-20T17:18:01.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21532025-03-20T17:18:01.750ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21542025-03-20T17:18:01.753ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21552025-03-20T17:18:01.754ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21562025-03-20T17:18:01.754ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
21572025-03-20T17:18:01.754ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
21582025-03-20T17:18:01.754ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21592025-03-20T17:18:01.754ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21602025-03-20T17:18:01.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21612025-03-20T17:18:01.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
21622025-03-20T17:18:01.755ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
21632025-03-20T17:18:01.755ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21642025-03-20T17:18:01.755ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
21652025-03-20T17:18:01.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21662025-03-20T17:18:01.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
21672025-03-20T17:18:01.755ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21682025-03-20T17:18:01.757ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21692025-03-20T17:18:01.758ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
21702025-03-20T17:18:01.758ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
21712025-03-20T17:18:01.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
21722025-03-20T17:18:01.758ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21732025-03-20T17:18:01.758ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21742025-03-20T17:18:01.759ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21752025-03-20T17:18:01.759ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21762025-03-20T17:18:01.759ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21772025-03-20T17:18:01.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21782025-03-20T17:18:01.764ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
21792025-03-20T17:18:01.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21802025-03-20T17:18:01.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21812025-03-20T17:18:01.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21822025-03-20T17:18:01.764ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
21832025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21842025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
21852025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21862025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21872025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21882025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21892025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
21902025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21912025-03-20T17:18:01.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21922025-03-20T17:18:01.766ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
21932025-03-20T17:18:01.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21942025-03-20T17:18:01.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21952025-03-20T17:18:01.770ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21962025-03-20T17:18:01.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21972025-03-20T17:18:01.771ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 83.048816ms iteration = 29 status = {"errors":[],"requests_created_ok":[]}
21982025-03-20T17:18:01.771ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21992025-03-20T17:18:01.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22002025-03-20T17:18:01.773ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 90.556145ms iteration = 85 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22012025-03-20T17:18:01.773ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22022025-03-20T17:18:01.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22032025-03-20T17:18:01.773ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
22042025-03-20T17:18:01.773ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
22052025-03-20T17:18:01.773ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 85.607983ms iteration = 15 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
22062025-03-20T17:18:01.777ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 89.529924ms iteration = 29 status = {"errors":[],"finish_invoked_ok":[]}
22072025-03-20T17:18:01.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
22082025-03-20T17:18:01.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22092025-03-20T17:18:01.783ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2915, tv_nsec: 242783587 } })
22102025-03-20T17:18:01.783ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
22112025-03-20T17:18:01.801ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
22122025-03-20T17:18:01.801ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 125.747272ms iteration = 85 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
22132025-03-20T17:18:01.813ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
22142025-03-20T17:18:01.813ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
22152025-03-20T17:18:01.813ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
22162025-03-20T17:18:01.813ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22172025-03-20T17:18:01.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22182025-03-20T17:18:01.815ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 127.314909ms iteration = 29 status = {"errors":[],"garbage_collect_requested":[]}
22192025-03-20T17:18:01.816ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 140.007081ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
22202025-03-20T17:18:01.816ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22212025-03-20T17:18:01.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22222025-03-20T17:18:01.817ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22232025-03-20T17:18:01.821ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22242025-03-20T17:18:01.821ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22252025-03-20T17:18:01.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22262025-03-20T17:18:01.822ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
22272025-03-20T17:18:01.822ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 146.548809ms iteration = 29 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
22282025-03-20T17:18:01.824ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
22292025-03-20T17:18:01.824ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22302025-03-20T17:18:01.824ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f420e403-4465-43f7-9f40-787b9fd2047c", "content-length": "26", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} })
22312025-03-20T17:18:01.824ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22322025-03-20T17:18:01.826ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22332025-03-20T17:18:01.826ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
22342025-03-20T17:18:01.826ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
22352025-03-20T17:18:01.827ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
22362025-03-20T17:18:01.827ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
22372025-03-20T17:18:01.827ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22382025-03-20T17:18:01.827ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
22392025-03-20T17:18:01.827ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22402025-03-20T17:18:01.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22412025-03-20T17:18:01.827ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "90921956-7dfd-4479-b499-b9e156d938d6", "content-length": "26", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} })
22422025-03-20T17:18:01.827ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22432025-03-20T17:18:01.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22442025-03-20T17:18:01.828ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22452025-03-20T17:18:01.828ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
22462025-03-20T17:18:01.829ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
22472025-03-20T17:18:01.829ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
22482025-03-20T17:18:01.829ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 153.137488ms iteration = 15 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
22492025-03-20T17:18:01.829ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22502025-03-20T17:18:01.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22512025-03-20T17:18:01.829ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
22522025-03-20T17:18:01.829ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 147.205746ms iteration = 29 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
22532025-03-20T17:18:01.830ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
22542025-03-20T17:18:01.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
22552025-03-20T17:18:01.830ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
22562025-03-20T17:18:01.831ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22572025-03-20T17:18:01.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22582025-03-20T17:18:01.832ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22592025-03-20T17:18:01.832ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
22602025-03-20T17:18:01.832ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
22612025-03-20T17:18:01.832ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22622025-03-20T17:18:01.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22632025-03-20T17:18:01.834ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
22642025-03-20T17:18:01.834ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22652025-03-20T17:18:01.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22662025-03-20T17:18:01.836ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22672025-03-20T17:18:01.836ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 59be64ba-8ced-4cf9-bc90-5b69ad6e044a, lookup_type: ById(59be64ba-8ced-4cf9-bc90-5b69ad6e044a) } result = Ok(())
22682025-03-20T17:18:01.836ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22692025-03-20T17:18:01.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22702025-03-20T17:18:01.837ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22712025-03-20T17:18:01.839ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
22722025-03-20T17:18:01.839ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22732025-03-20T17:18:01.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22742025-03-20T17:18:01.840ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
22752025-03-20T17:18:01.840ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 163.87646ms iteration = 29 status = 0
22762025-03-20T17:18:01.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22772025-03-20T17:18:01.841ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22782025-03-20T17:18:01.841ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22792025-03-20T17:18:01.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22802025-03-20T17:18:01.842ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 153.928037ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
22812025-03-20T17:18:01.845ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
22822025-03-20T17:18:01.862ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22832025-03-20T17:18:01.866ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
22842025-03-20T17:18:01.866ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22852025-03-20T17:18:01.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22862025-03-20T17:18:01.872ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
22872025-03-20T17:18:01.872ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d44f50e8-2d0f-4b3e-9e56-9e88b49097cc dns_zone_name = oxide.test found_so_far = 0 version = 2
22882025-03-20T17:18:01.872ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22892025-03-20T17:18:01.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22902025-03-20T17:18:01.873ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 185.269973ms iteration = 29 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
22912025-03-20T17:18:01.873ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22922025-03-20T17:18:01.874ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22932025-03-20T17:18:01.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22942025-03-20T17:18:01.877ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
22952025-03-20T17:18:01.877ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22962025-03-20T17:18:01.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22972025-03-20T17:18:01.879ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d44f50e8-2d0f-4b3e-9e56-9e88b49097cc dns_zone_name = oxide.test found_so_far = 1 version = 2
22982025-03-20T17:18:01.879ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
22992025-03-20T17:18:01.879ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-20 17:04:53.696227 UTC dns_group = external generation = 2
23002025-03-20T17:18:01.879ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 203.279402ms iteration = 15 status = {"generation":2}
23012025-03-20T17:18:01.885ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23022025-03-20T17:18:01.885ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
23032025-03-20T17:18:01.885ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 198.713944ms iteration = 29 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
23042025-03-20T17:18:01.889ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-20T17:04:00.028646Z, time_modified: 2025-03-20T17:04:00.028646Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ca06964a-7ce0-46e9-890b-0b57019b1afc, db_silo: Silo { identity: SiloIdentity { id: ca06964a-7ce0-46e9-890b-0b57019b1afc, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-20T17:04:53.538118Z, time_modified: 2025-03-20T17:04:53.538118Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ca06964a-7ce0-46e9-890b-0b57019b1afc, db_silo: Silo { identity: SiloIdentity { id: ca06964a-7ce0-46e9-890b-0b57019b1afc, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-20T17:04:53.538118Z, time_modified: 2025-03-20T17:04:53.538118Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63" }] }) }
23052025-03-20T17:18:01.889ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 213.356197ms iteration = 15 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ca06964a-7ce0-46e9-890b-0b57019b1afc","tls_certs":["450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63"]}},"default_endpoint":{"silo_id":"ca06964a-7ce0-46e9-890b-0b57019b1afc","tls_certs":["450f2acd645e06882b9bbb5b71b6b67fb732ae4f47860d1ed9258f9ef11a3c63"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
23062025-03-20T17:18:01.904ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
23072025-03-20T17:18:01.904ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 02f70c4b-db30-4840-b4d1-314d13a3510d dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
23082025-03-20T17:18:01.904ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23092025-03-20T17:18:01.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23102025-03-20T17:18:01.909ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23112025-03-20T17:18:01.914ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23122025-03-20T17:18:01.914ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23132025-03-20T17:18:01.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23142025-03-20T17:18:01.917ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 02f70c4b-db30-4840-b4d1-314d13a3510d dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
23152025-03-20T17:18:01.917ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
23162025-03-20T17:18:01.917ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-20 17:04:53.071389 UTC dns_group = internal generation = 1
23172025-03-20T17:18:01.917ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 241.661563ms iteration = 15 status = {"generation":1}
23182025-03-20T17:18:01.922ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
23192025-03-20T17:18:01.922ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 235.72916ms iteration = 15 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
23202025-03-20T17:18:01.934ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 zone_address = fd00:1122:3344:101::2
23212025-03-20T17:18:01.934ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23222025-03-20T17:18:01.935ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "acf31542-9c28-490d-ac0b-442f92efbfd9", "content-length": "26", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} })
23232025-03-20T17:18:01.935ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23242025-03-20T17:18:01.937ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
23252025-03-20T17:18:01.967ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 291.574694ms iteration = 31 status = {"physical_disks_added":0}
23262025-03-20T17:18:01.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23272025-03-20T17:18:01.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23282025-03-20T17:18:01.992ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
23292025-03-20T17:18:01.992ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c578814-2298-4321-a38f-8d91d4041495", "content-length": "2", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} })
23302025-03-20T17:18:01.994ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
23312025-03-20T17:18:01.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
23322025-03-20T17:18:02.001ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
23332025-03-20T17:18:02.001ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "6ccd0460-32c1-4ec3-aa25-87441a501f40", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23342025-03-20T17:18:02.001ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23352025-03-20T17:18:02.004ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "d555daf3-c19f-41f6-bf5b-f3c314b66edf", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23362025-03-20T17:18:02.004ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23372025-03-20T17:18:02.005ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23382025-03-20T17:18:02.005ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 329.335337ms iteration = 17 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
23392025-03-20T17:18:02.013ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
23402025-03-20T17:18:02.020ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 344.787818ms iteration = 29 status = {}
23412025-03-20T17:18:02.041ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
23422025-03-20T17:18:02.041ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
23432025-03-20T17:18:02.042ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23442025-03-20T17:18:02.043ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 367.473806ms iteration = 29 status = null
23452025-03-20T17:18:02.044ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23462025-03-20T17:18:02.044ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23472025-03-20T17:18:02.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23482025-03-20T17:18:02.046ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23492025-03-20T17:18:02.048ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23502025-03-20T17:18:02.048ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23512025-03-20T17:18:02.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23522025-03-20T17:18:02.069ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d5491a-3c6c-46cb-b179-3fcbf65eae47", "content-length": "372", "date": "Thu, 20 Mar 2025 17:18:02 GMT"} })
23532025-03-20T17:18:02.069ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23542025-03-20T17:18:02.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23552025-03-20T17:18:02.091ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
23562025-03-20T17:18:02.091ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "684e1241-cc4e-492a-8650-dc8f2e53d984", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23572025-03-20T17:18:02.091ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23582025-03-20T17:18:02.093ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "6686e084-6490-4f64-908f-45558a2b50f8", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23592025-03-20T17:18:02.093ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23602025-03-20T17:18:02.095ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "dd13e885-6827-48a9-ba16-af45d8f4aea3", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23612025-03-20T17:18:02.095ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23622025-03-20T17:18:02.097ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23632025-03-20T17:18:02.097ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 421.291928ms iteration = 17 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
23642025-03-20T17:18:02.109ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
23652025-03-20T17:18:02.109ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
23662025-03-20T17:18:02.109ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ee2de7fc-7bc8-4374-a175-2a2e3efe6363", "date": "Thu, 20 Mar 2025 17:18:02 GMT"} })
23672025-03-20T17:18:02.109ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23682025-03-20T17:18:02.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23692025-03-20T17:18:02.117ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
23702025-03-20T17:18:02.117ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "29e6367d-e50b-4e66-bce1-24dec12e11aa", "content-length": "691", "date": "Thu, 20 Mar 2025 17:18:02 GMT"} })
23712025-03-20T17:18:02.117ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
23722025-03-20T17:18:02.117ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
23732025-03-20T17:18:02.117ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
23742025-03-20T17:18:02.118ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 442.497819ms iteration = 29 status = {}
23752025-03-20T17:18:02.120ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23762025-03-20T17:18:02.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23772025-03-20T17:18:02.122ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23782025-03-20T17:18:02.122ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23792025-03-20T17:18:02.126ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
23802025-03-20T17:18:02.126ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
23812025-03-20T17:18:02.126ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "8635b9b2-0657-40c9-9ed3-9f56af0f2fde", "date": "Thu, 20 Mar 2025 17:18:02 GMT"} })
23822025-03-20T17:18:02.128ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 441.67612ms iteration = 29 status = {}
23832025-03-20T17:18:02.158ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23842025-03-20T17:18:02.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23852025-03-20T17:18:02.160ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
23862025-03-20T17:18:02.160ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23872025-03-20T17:18:02.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23882025-03-20T17:18:02.170ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
23892025-03-20T17:18:02.171ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "4324c134-b5f1-4302-a9bb-62aa1a0c383b", "content-length": "134", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} })
23902025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
23912025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
23922025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
23932025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
23942025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23952025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23962025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23972025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
23982025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23992025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24002025-03-20T17:18:02.171ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
24012025-03-20T17:18:02.171ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24022025-03-20T17:18:02.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24032025-03-20T17:18:02.174ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
24042025-03-20T17:18:02.174ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e9f5281-f11c-4fa6-9026-522a8c7d13e8", "content-length": "2", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} })
24052025-03-20T17:18:02.174ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
24062025-03-20T17:18:02.174ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "355795e1-da14-46a1-ae04-de7019f5426a", "content-length": "52", "date": "Thu, 20 Mar 2025 17:18:01 GMT"} })
24072025-03-20T17:18:02.174ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
24082025-03-20T17:18:02.174ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24092025-03-20T17:18:02.273ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "225fc7c3-576a-4d1e-afe1-86ad48b475f6", "date": "Thu, 20 Mar 2025 17:18:02 GMT"} })
24102025-03-20T17:18:02.273ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
24112025-03-20T17:18:02.273ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24122025-03-20T17:18:02.279ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0b3a6f2-672a-434f-a122-56313b4a20dd", "content-length": "682", "date": "Thu, 20 Mar 2025 17:18:02 GMT"} })
24132025-03-20T17:18:02.279ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24142025-03-20T17:18:02.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24152025-03-20T17:18:02.283ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24162025-03-20T17:18:02.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24172025-03-20T17:18:02.287ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24182025-03-20T17:18:02.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24192025-03-20T17:18:02.289ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24202025-03-20T17:18:02.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24212025-03-20T17:18:02.298ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 622.448028ms iteration = 29 status = {}
24222025-03-20T17:18:04.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 308 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = f73cf74e-2520-4fbf-9ebf-9465174f10f6 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
24232025-03-20T17:18:11.675ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 86 reason = Timeout
24242025-03-20T17:18:11.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24252025-03-20T17:18:11.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24262025-03-20T17:18:11.682ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 86 reason = Timeout
24272025-03-20T17:18:11.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24282025-03-20T17:18:11.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24292025-03-20T17:18:11.706ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24302025-03-20T17:18:11.709ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24312025-03-20T17:18:11.709ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24322025-03-20T17:18:11.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24332025-03-20T17:18:11.709ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24342025-03-20T17:18:11.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24352025-03-20T17:18:11.711ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24362025-03-20T17:18:11.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24372025-03-20T17:18:11.744ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24382025-03-20T17:18:11.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
24392025-03-20T17:18:11.747ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24402025-03-20T17:18:11.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24412025-03-20T17:18:11.747ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.884967ms iteration = 86 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24422025-03-20T17:18:11.756ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
24432025-03-20T17:18:11.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 81.050706ms iteration = 86 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
24442025-03-20T17:18:14.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 321 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 9f22ea81-2caf-4100-9600-909ccef1b55b response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
24452025-03-20T17:18:21.676ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 87 reason = Timeout
24462025-03-20T17:18:21.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24472025-03-20T17:18:21.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24482025-03-20T17:18:21.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24492025-03-20T17:18:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24502025-03-20T17:18:21.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24512025-03-20T17:18:21.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24522025-03-20T17:18:21.683ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24532025-03-20T17:18:21.683ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 87 reason = Timeout
24542025-03-20T17:18:21.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24552025-03-20T17:18:21.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24562025-03-20T17:18:21.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24572025-03-20T17:18:21.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24582025-03-20T17:18:21.685ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24592025-03-20T17:18:21.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24602025-03-20T17:18:21.687ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
24612025-03-20T17:18:21.687ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24622025-03-20T17:18:21.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24632025-03-20T17:18:21.695ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
24642025-03-20T17:18:21.695ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.638435ms iteration = 87 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
24652025-03-20T17:18:21.715ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.776632ms iteration = 87 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24662025-03-20T17:18:24.945ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 321 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 29fb9f3f-daf3-4fe1-a181-4f653a0305fa response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
24672025-03-20T17:18:31.675ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 88 reason = Timeout
24682025-03-20T17:18:31.675ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 30 reason = Timeout
24692025-03-20T17:18:31.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24702025-03-20T17:18:31.676ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24712025-03-20T17:18:31.676ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 30 reason = Timeout
24722025-03-20T17:18:31.676ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 30 reason = Timeout
24732025-03-20T17:18:31.676ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
24742025-03-20T17:18:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24752025-03-20T17:18:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
24762025-03-20T17:18:31.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24772025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 30 reason = Timeout
24782025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
24792025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 32 reason = Timeout
24802025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24812025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 30 reason = Timeout
24822025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 30 reason = Timeout
24832025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24842025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24852025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 30 reason = Timeout
24862025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24872025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 30 reason = Timeout
24882025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24892025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24902025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
24912025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24922025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24932025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24942025-03-20T17:18:31.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24952025-03-20T17:18:31.680ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2945, tv_nsec: 139860134 } })
24962025-03-20T17:18:31.681ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24972025-03-20T17:18:31.681ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
24982025-03-20T17:18:31.681ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2945, tv_nsec: 140400559 } })
24992025-03-20T17:18:31.681ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25002025-03-20T17:18:31.681ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
25012025-03-20T17:18:31.682ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.70871ms iteration = 30 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
25022025-03-20T17:18:31.682ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 30 reason = Timeout
25032025-03-20T17:18:31.682ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25042025-03-20T17:18:31.683ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 88 reason = Timeout
25052025-03-20T17:18:31.683ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25062025-03-20T17:18:31.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25072025-03-20T17:18:31.683ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25082025-03-20T17:18:31.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25092025-03-20T17:18:31.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25102025-03-20T17:18:31.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
25112025-03-20T17:18:31.684ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25122025-03-20T17:18:31.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25132025-03-20T17:18:31.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25142025-03-20T17:18:31.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25152025-03-20T17:18:31.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25162025-03-20T17:18:31.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
25172025-03-20T17:18:31.686ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25182025-03-20T17:18:31.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25192025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
25202025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25212025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 30 reason = Timeout
25222025-03-20T17:18:31.687ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
25232025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25242025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 30 reason = Timeout
25252025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
25262025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25272025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25282025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25292025-03-20T17:18:31.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25302025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 30 reason = Timeout
25312025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
25322025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 30 reason = Timeout
25332025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
25342025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 30 reason = Timeout
25352025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 30 reason = Timeout
25362025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
25372025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
25382025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 30 reason = Timeout
25392025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25402025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
25412025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
25422025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
25432025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
25442025-03-20T17:18:31.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25452025-03-20T17:18:31.692ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25462025-03-20T17:18:31.692ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25472025-03-20T17:18:31.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25482025-03-20T17:18:31.693ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25492025-03-20T17:18:31.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25502025-03-20T17:18:31.694ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-59be64ba-8ced-4cf9-bc90-5b69ad6e044a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 2945, tv_nsec: 153911030 } })
25512025-03-20T17:18:31.695ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
25522025-03-20T17:18:31.695ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
25532025-03-20T17:18:31.695ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25542025-03-20T17:18:31.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25552025-03-20T17:18:31.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25562025-03-20T17:18:31.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25572025-03-20T17:18:31.699ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 9.793283ms iteration = 30 status = {"errors":[],"garbage_collect_requested":[]}
25582025-03-20T17:18:31.733ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
25592025-03-20T17:18:31.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
25602025-03-20T17:18:31.733ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25612025-03-20T17:18:31.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25622025-03-20T17:18:31.736ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25632025-03-20T17:18:31.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25642025-03-20T17:18:31.743ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25652025-03-20T17:18:31.743ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
25662025-03-20T17:18:31.743ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25672025-03-20T17:18:31.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25682025-03-20T17:18:31.744ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25692025-03-20T17:18:31.745ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25702025-03-20T17:18:31.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25712025-03-20T17:18:31.747ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25722025-03-20T17:18:31.747ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
25732025-03-20T17:18:31.747ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25742025-03-20T17:18:31.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
25752025-03-20T17:18:31.747ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25762025-03-20T17:18:31.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25772025-03-20T17:18:31.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25782025-03-20T17:18:31.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25792025-03-20T17:18:31.750ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
25802025-03-20T17:18:31.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
25812025-03-20T17:18:31.751ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25822025-03-20T17:18:31.752ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25832025-03-20T17:18:31.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25842025-03-20T17:18:31.753ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 64.430632ms iteration = 30 status = {"errors":[],"requests_created_ok":[]}
25852025-03-20T17:18:31.753ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
25862025-03-20T17:18:31.753ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25872025-03-20T17:18:31.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25882025-03-20T17:18:31.754ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
25892025-03-20T17:18:31.754ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25902025-03-20T17:18:31.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25912025-03-20T17:18:31.756ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25922025-03-20T17:18:31.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25932025-03-20T17:18:31.757ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25942025-03-20T17:18:31.758ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25952025-03-20T17:18:31.758ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 69.327743ms iteration = 30 status = {"errors":[],"finish_invoked_ok":[]}
25962025-03-20T17:18:31.760ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
25972025-03-20T17:18:31.760ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25982025-03-20T17:18:31.760ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 77.838292ms iteration = 88 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25992025-03-20T17:18:31.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26002025-03-20T17:18:31.761ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
26012025-03-20T17:18:31.761ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26022025-03-20T17:18:31.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26032025-03-20T17:18:31.770ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
26042025-03-20T17:18:31.770ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 87.419432ms iteration = 30 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
26052025-03-20T17:18:31.771ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
26062025-03-20T17:18:31.771ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26072025-03-20T17:18:31.771ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "38a5255f-9ecb-4264-b97f-7405c01bf05c", "content-length": "26", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26082025-03-20T17:18:31.771ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26092025-03-20T17:18:31.773ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26102025-03-20T17:18:31.773ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
26112025-03-20T17:18:31.773ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 97.75183ms iteration = 88 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
26122025-03-20T17:18:31.774ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
26132025-03-20T17:18:31.774ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26142025-03-20T17:18:31.774ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d21f27ff-c0df-4f3c-812f-39ab425c4ae6", "content-length": "26", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26152025-03-20T17:18:31.774ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26162025-03-20T17:18:31.775ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26172025-03-20T17:18:31.778ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 101.480759ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
26182025-03-20T17:18:31.783ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26192025-03-20T17:18:31.784ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 95.421896ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
26202025-03-20T17:18:31.785ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 zone_address = fd00:1122:3344:101::2
26212025-03-20T17:18:31.785ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26222025-03-20T17:18:31.786ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c4ac5e-6a01-4235-8264-1001e38d99eb", "content-length": "26", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26232025-03-20T17:18:31.786ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26242025-03-20T17:18:31.786ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
26252025-03-20T17:18:31.786ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
26262025-03-20T17:18:31.786ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 99.165654ms iteration = 30 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
26272025-03-20T17:18:31.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 59be64ba-8ced-4cf9-bc90-5b69ad6e044a, lookup_type: ById(59be64ba-8ced-4cf9-bc90-5b69ad6e044a) } result = Ok(())
26282025-03-20T17:18:31.787ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26292025-03-20T17:18:31.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26302025-03-20T17:18:31.788ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
26312025-03-20T17:18:31.789ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 112.1716ms iteration = 30 status = 0
26322025-03-20T17:18:31.795ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26332025-03-20T17:18:31.796ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26342025-03-20T17:18:31.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26352025-03-20T17:18:31.798ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
26362025-03-20T17:18:31.798ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26372025-03-20T17:18:31.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26382025-03-20T17:18:31.800ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26392025-03-20T17:18:31.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26402025-03-20T17:18:31.828ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 138.780548ms iteration = 30 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
26412025-03-20T17:18:31.829ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
26422025-03-20T17:18:31.829ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 152.946136ms iteration = 30 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
26432025-03-20T17:18:31.871ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
26442025-03-20T17:18:31.873ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb42e438-e8d9-46f5-b547-dadd9c0cf59b", "content-length": "2", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26452025-03-20T17:18:31.875ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 198.752385ms iteration = 32 status = {"physical_disks_added":0}
26462025-03-20T17:18:31.875ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26472025-03-20T17:18:31.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26482025-03-20T17:18:31.878ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 202.126749ms iteration = 30 status = null
26492025-03-20T17:18:31.878ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 202.320222ms iteration = 30 status = {}
26502025-03-20T17:18:31.915ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26512025-03-20T17:18:31.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26522025-03-20T17:18:31.917ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26532025-03-20T17:18:31.919ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
26542025-03-20T17:18:31.919ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26552025-03-20T17:18:31.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26562025-03-20T17:18:31.921ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26572025-03-20T17:18:31.922ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
26582025-03-20T17:18:31.922ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
26592025-03-20T17:18:31.923ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
26602025-03-20T17:18:31.923ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26612025-03-20T17:18:31.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26622025-03-20T17:18:31.933ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "174ed0df-9c41-4a24-9748-4100074ed9d4", "content-length": "372", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26632025-03-20T17:18:31.933ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26642025-03-20T17:18:31.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26652025-03-20T17:18:31.975ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
26662025-03-20T17:18:31.975ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
26672025-03-20T17:18:31.976ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "fc5aae21-421e-4a90-8746-7f950b377f29", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26682025-03-20T17:18:31.976ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26692025-03-20T17:18:31.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26702025-03-20T17:18:31.978ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26712025-03-20T17:18:31.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26722025-03-20T17:18:31.981ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26732025-03-20T17:18:31.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26742025-03-20T17:18:31.989ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
26752025-03-20T17:18:31.989ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "15a514fe-8fcf-400e-8b75-f816b55707c8", "content-length": "691", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26762025-03-20T17:18:31.989ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
26772025-03-20T17:18:31.989ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
26782025-03-20T17:18:31.989ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
26792025-03-20T17:18:31.990ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 313.649014ms iteration = 30 status = {}
26802025-03-20T17:18:31.992ZINFOnexus (BackgroundTasks): VPC route manager sled 59be64ba-8ced-4cf9-bc90-5b69ad6e044a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
26812025-03-20T17:18:31.992ZDEBGnexus (BackgroundTasks): client request SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
26822025-03-20T17:18:31.996ZDEBGnexus (BackgroundTasks): client response SledAgent = 59be64ba-8ced-4cf9-bc90-5b69ad6e044a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "707bcaf7-bcc4-4329-9c3c-707a4cc41e9a", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26832025-03-20T17:18:31.998ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 311.569362ms iteration = 30 status = {}
26842025-03-20T17:18:32.002ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26852025-03-20T17:18:32.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26862025-03-20T17:18:32.004ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 430461af-39f0-4c9b-b73e-c8f180a5058f, rack_id: 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(6edeb06d-6be6-4b7a-ad12-74c22a2e9b97) } rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
26872025-03-20T17:18:32.004ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26882025-03-20T17:18:32.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26892025-03-20T17:18:32.015ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
26902025-03-20T17:18:32.016ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "01192d7b-0597-46e8-a260-ca31ef68432a", "content-length": "134", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
26912025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
26922025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 6edeb06d-6be6-4b7a-ad12-74c22a2e9b97
26932025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
26942025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
26952025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26962025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26972025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26982025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
26992025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
27002025-03-20T17:18:32.016ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
27012025-03-20T17:18:32.017ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1
27022025-03-20T17:18:32.017ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27032025-03-20T17:18:32.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27042025-03-20T17:18:32.019ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
27052025-03-20T17:18:32.019ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "50419cd5-7ee6-4ec4-8574-01e126cb3068", "content-length": "2", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
27062025-03-20T17:18:32.019ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
27072025-03-20T17:18:32.020ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d20c504-c0f4-451f-bc33-bef18507b394", "content-length": "52", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
27082025-03-20T17:18:32.020ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch_location = Switch0
27092025-03-20T17:18:32.020ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
27102025-03-20T17:18:32.199ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4c8364a8-b2fa-4c00-bb47-9d9cef67fc15", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
27112025-03-20T17:18:32.199ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 switch = Switch0
27122025-03-20T17:18:32.199ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
27132025-03-20T17:18:32.200ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 0f172a51-e9dd-4435-92ca-55eb7bbb2bb1 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "029abd0a-85ed-422f-a2a7-65f5714c396f", "content-length": "682", "date": "Thu, 20 Mar 2025 17:18:31 GMT"} })
27142025-03-20T17:18:32.200ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27152025-03-20T17:18:32.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27162025-03-20T17:18:32.203ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27172025-03-20T17:18:32.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27182025-03-20T17:18:32.207ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27192025-03-20T17:18:32.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27202025-03-20T17:18:32.209ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27212025-03-20T17:18:32.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27222025-03-20T17:18:32.217ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 540.817245ms iteration = 30 status = {}
27232025-03-20T17:18:34.945ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 307 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = c3b3a49f-42ab-4276-a773-37a0e345f6d1 response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
27242025-03-20T17:18:41.676ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 89 reason = Timeout
27252025-03-20T17:18:41.676ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27262025-03-20T17:18:41.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27272025-03-20T17:18:41.678ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27282025-03-20T17:18:41.682ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27292025-03-20T17:18:41.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27302025-03-20T17:18:41.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27312025-03-20T17:18:41.683ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 89 reason = Timeout
27322025-03-20T17:18:41.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27332025-03-20T17:18:41.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27342025-03-20T17:18:41.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27352025-03-20T17:18:41.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27362025-03-20T17:18:41.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27372025-03-20T17:18:41.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
27382025-03-20T17:18:41.688ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27392025-03-20T17:18:41.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27402025-03-20T17:18:41.697ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
27412025-03-20T17:18:41.697ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.181391ms iteration = 89 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
27422025-03-20T17:18:41.712ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27432025-03-20T17:18:41.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27442025-03-20T17:18:41.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.439471ms iteration = 89 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27452025-03-20T17:18:44.936ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:46664
27462025-03-20T17:18:44.936ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46664 req_id = 67927921-4bfe-46cf-83a9-84e35ac5d7cd roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
27472025-03-20T17:18:44.937ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46664 req_id = 67927921-4bfe-46cf-83a9-84e35ac5d7cd resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
27482025-03-20T17:18:44.940ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3486 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46664 req_id = 67927921-4bfe-46cf-83a9-84e35ac5d7cd response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?sort_by=id_ascending
27492025-03-20T17:18:44.940ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46664 req_id = 55da165a-ccf4-420e-9f46-2af168bf8d3a roles = RoleSet { roles: {} } uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
27502025-03-20T17:18:44.940ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46664 req_id = 55da165a-ccf4-420e-9f46-2af168bf8d3a resource = Database result = Ok(()) uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
27512025-03-20T17:18:44.943ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2490 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46664 req_id = 55da165a-ccf4-420e-9f46-2af168bf8d3a response_code = 200 uri = /metrics/collectors/6125e0e7-3df0-4e3f-a39d-9936863e5edc/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2ExNzRhYS0wYjUwLTRiMDEtYmUxMi05MGRhODUwNTM3YWYifX0%3D
27522025-03-20T17:18:44.944ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = 188e1867-eaee-498c-ad3d-fb596305641c response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c
27532025-03-20T17:18:51.677ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 90 reason = Timeout
27542025-03-20T17:18:51.677ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27552025-03-20T17:18:51.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27562025-03-20T17:18:51.679ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27572025-03-20T17:18:51.682ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27582025-03-20T17:18:51.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27592025-03-20T17:18:51.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27602025-03-20T17:18:51.683ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 90 reason = Timeout
27612025-03-20T17:18:51.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27622025-03-20T17:18:51.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27632025-03-20T17:18:51.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27642025-03-20T17:18:51.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27652025-03-20T17:18:51.685ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27662025-03-20T17:18:51.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27672025-03-20T17:18:51.686ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27682025-03-20T17:18:51.687ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.895131ms iteration = 90 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27692025-03-20T17:18:51.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 24f105be-07da-48d8-9788-12335c722319, lookup_type: ById(24f105be-07da-48d8-9788-12335c722319) } result = Ok(())
27702025-03-20T17:18:51.688ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27712025-03-20T17:18:51.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27722025-03-20T17:18:51.697ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 24f105be-07da-48d8-9788-12335c722319 original_time_created = 2025-03-20 17:03:54.301101 UTC target_id = 24f105be-07da-48d8-9788-12335c722319 time_created = 2025-03-20 17:03:54.301101 UTC
27732025-03-20T17:18:51.697ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.1271ms iteration = 90 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"24f105be-07da-48d8-9788-12335c722319","time_created":"2025-03-20T17:03:54.301101Z"}
27742025-03-20T17:18:54.945ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 370 local_addr = [fd00:1122:3344:101::a]:56049 method = GET remote_addr = [fd00:1122:3344:101::d]:59376 req_id = e7360672-cdaf-457b-99a3-25eeec5983bf response_code = 200 uri = /4bdaf2c6-9ecf-45b4-b193-a6e822d5855c