Max level shown:
12025-03-17T16:45:05.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 956d798e-e4a1-431f-ae0b-66fb8e3d1368 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
22025-03-17T16:45:05.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 956d798e-e4a1-431f-ae0b-66fb8e3d1368 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
32025-03-17T16:45:05.128ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2877 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 956d798e-e4a1-431f-ae0b-66fb8e3d1368 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
42025-03-17T16:45:05.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = b9b91282-a4b7-4dd8-b101-a48dd7060739 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
52025-03-17T16:45:05.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = b9b91282-a4b7-4dd8-b101-a48dd7060739 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
62025-03-17T16:45:05.131ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2739 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = b9b91282-a4b7-4dd8-b101-a48dd7060739 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
72025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
82025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
92025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
102025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
112025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
122025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
132025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
142025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
152025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
162025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
172025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
182025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
202025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
212025-03-17T16:45:06.594ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:35:06.594376743 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
222025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
232025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
242025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
252025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
262025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
272025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
282025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
292025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
302025-03-17T16:45:06.594ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
312025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.271\u{b5}s iteration = 6 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
322025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 5 reason = Timeout
332025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 5 reason = Timeout
342025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
352025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
362025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
372025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
382025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
392025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
402025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 9 reason = Timeout
412025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
422025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
432025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 25 reason = Timeout
442025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
452025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
462025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
472025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 5 reason = Timeout
482025-03-17T16:45:06.594ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
492025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
502025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 9 reason = Timeout
512025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
522025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 9 reason = Timeout
532025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
542025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
552025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
562025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 5 reason = Timeout
572025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
582025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 9 reason = Timeout
592025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
602025-03-17T16:45:06.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
612025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 9 reason = Timeout
622025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
632025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
642025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
652025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 9 reason = Timeout
662025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
672025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
682025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 9 reason = Timeout
692025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
702025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
712025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
722025-03-17T16:45:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
732025-03-17T16:45:06.595ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 3993, tv_nsec: 856697332 } })
742025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
752025-03-17T16:45:06.595ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 3993, tv_nsec: 856725323 } })
762025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
772025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
782025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
792025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
802025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
812025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
822025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
832025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
842025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
852025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
862025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
872025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
882025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
892025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
902025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
912025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
922025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
932025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
942025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
952025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
962025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.101914ms iteration = 5 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
972025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
982025-03-17T16:45:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.333848ms iteration = 5 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
992025-03-17T16:45:06.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1002025-03-17T16:45:06.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1012025-03-17T16:45:06.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1022025-03-17T16:45:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1032025-03-17T16:45:06.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3993, tv_nsec: 859146055 } })
1042025-03-17T16:45:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
1052025-03-17T16:45:06.598ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.77495ms iteration = 9 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
1062025-03-17T16:45:06.598ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:35:06.594376743 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
1072025-03-17T16:45:06.598ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.353983ms iteration = 5 status = {"expiration":"2025-03-17T16:35:06.594376743Z","pruned":[]}
1082025-03-17T16:45:06.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1092025-03-17T16:45:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1102025-03-17T16:45:06.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1112025-03-17T16:45:06.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1122025-03-17T16:45:06.606ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1132025-03-17T16:45:06.606ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 11.986507ms iteration = 5 status = {"errors":[],"found_port_ok":[]}
1142025-03-17T16:45:06.607ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1152025-03-17T16:45:06.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1162025-03-17T16:45:06.609ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1172025-03-17T16:45:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1182025-03-17T16:45:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1192025-03-17T16:45:06.611ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1202025-03-17T16:45:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1212025-03-17T16:45:06.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 9 reason = Timeout
1222025-03-17T16:45:06.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
1232025-03-17T16:45:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1242025-03-17T16:45:06.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 9 reason = Timeout
1252025-03-17T16:45:06.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
1262025-03-17T16:45:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1272025-03-17T16:45:06.619ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1282025-03-17T16:45:06.621ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1292025-03-17T16:45:06.623ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1302025-03-17T16:45:06.623ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1312025-03-17T16:45:06.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1322025-03-17T16:45:06.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1332025-03-17T16:45:06.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1342025-03-17T16:45:06.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1352025-03-17T16:45:06.640ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1362025-03-17T16:45:06.642ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
1372025-03-17T16:45:06.642ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
1382025-03-17T16:45:06.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
1392025-03-17T16:45:06.645ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1402025-03-17T16:45:06.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1412025-03-17T16:45:06.649ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.887706ms iteration = 9 status = {"errors":[],"garbage_collect_requested":[]}
1422025-03-17T16:45:06.654ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1432025-03-17T16:45:06.654ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1442025-03-17T16:45:06.654ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1452025-03-17T16:45:06.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1462025-03-17T16:45:06.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1472025-03-17T16:45:06.656ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1482025-03-17T16:45:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1492025-03-17T16:45:06.657ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
1502025-03-17T16:45:06.657ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
1512025-03-17T16:45:06.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1522025-03-17T16:45:06.657ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 62.539149ms iteration = 5 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
1532025-03-17T16:45:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1542025-03-17T16:45:06.658ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1552025-03-17T16:45:06.658ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1562025-03-17T16:45:06.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1572025-03-17T16:45:06.658ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1582025-03-17T16:45:06.659ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1592025-03-17T16:45:06.660ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1602025-03-17T16:45:06.661ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1612025-03-17T16:45:06.661ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1622025-03-17T16:45:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1632025-03-17T16:45:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1642025-03-17T16:45:06.662ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1652025-03-17T16:45:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1662025-03-17T16:45:06.663ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1672025-03-17T16:45:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1682025-03-17T16:45:06.663ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1692025-03-17T16:45:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1702025-03-17T16:45:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1712025-03-17T16:45:06.666ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1722025-03-17T16:45:06.666ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1732025-03-17T16:45:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
1742025-03-17T16:45:06.670ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1752025-03-17T16:45:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1762025-03-17T16:45:06.670ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1772025-03-17T16:45:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1782025-03-17T16:45:06.671ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
1792025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
1802025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
1812025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
1822025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1832025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1842025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1852025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1862025-03-17T16:45:06.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1872025-03-17T16:45:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1882025-03-17T16:45:06.676ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3993, tv_nsec: 938550625 } })
1892025-03-17T16:45:06.677ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
1902025-03-17T16:45:06.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1912025-03-17T16:45:06.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
1922025-03-17T16:45:06.681ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1932025-03-17T16:45:06.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1942025-03-17T16:45:06.683ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1952025-03-17T16:45:06.684ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
1962025-03-17T16:45:06.684ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 90.389096ms iteration = 9 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
1972025-03-17T16:45:06.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
1982025-03-17T16:45:06.688ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1992025-03-17T16:45:06.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2002025-03-17T16:45:06.689ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 3993, tv_nsec: 950926620 } })
2012025-03-17T16:45:06.689ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
2022025-03-17T16:45:06.690ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2032025-03-17T16:45:06.690ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
2042025-03-17T16:45:06.690ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
2052025-03-17T16:45:06.690ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
2062025-03-17T16:45:06.690ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2072025-03-17T16:45:06.691ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2082025-03-17T16:45:06.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2092025-03-17T16:45:06.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2102025-03-17T16:45:06.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
2112025-03-17T16:45:06.693ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2122025-03-17T16:45:06.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2132025-03-17T16:45:06.694ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2142025-03-17T16:45:06.694ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
2152025-03-17T16:45:06.694ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 100.529693ms iteration = 25 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
2162025-03-17T16:45:06.697ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2172025-03-17T16:45:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2182025-03-17T16:45:06.699ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
2192025-03-17T16:45:06.699ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2202025-03-17T16:45:06.699ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
2212025-03-17T16:45:06.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2222025-03-17T16:45:06.703ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 86.590775ms iteration = 9 status = {"errors":[],"requests_created_ok":[]}
2232025-03-17T16:45:06.703ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 109.001144ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2242025-03-17T16:45:06.703ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 86.956682ms iteration = 9 status = {"errors":[],"finish_invoked_ok":[]}
2252025-03-17T16:45:06.704ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
2262025-03-17T16:45:06.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
2272025-03-17T16:45:06.708ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
2282025-03-17T16:45:06.708ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2292025-03-17T16:45:06.708ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "16237aea-e61f-4641-9233-66ab8a496d09", "content-length": "26", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
2302025-03-17T16:45:06.708ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2312025-03-17T16:45:06.709ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
2322025-03-17T16:45:06.709ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
2332025-03-17T16:45:06.709ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
2342025-03-17T16:45:06.709ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 115.391711ms iteration = 5 status = {"generation":2}
2352025-03-17T16:45:06.709ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2362025-03-17T16:45:06.709ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
2372025-03-17T16:45:06.710ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2382025-03-17T16:45:06.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2392025-03-17T16:45:06.710ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2402025-03-17T16:45:06.714ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 119.699114ms iteration = 25 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2412025-03-17T16:45:06.716ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2422025-03-17T16:45:06.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2432025-03-17T16:45:06.716ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2442025-03-17T16:45:06.717ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2452025-03-17T16:45:06.717ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
2462025-03-17T16:45:06.717ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
2472025-03-17T16:45:06.717ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
2482025-03-17T16:45:06.717ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2492025-03-17T16:45:06.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2502025-03-17T16:45:06.720ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
2512025-03-17T16:45:06.720ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
2522025-03-17T16:45:06.720ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 125.710052ms iteration = 5 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
2532025-03-17T16:45:06.720ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2542025-03-17T16:45:06.720ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2552025-03-17T16:45:06.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2562025-03-17T16:45:06.721ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
2572025-03-17T16:45:06.721ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2582025-03-17T16:45:06.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2592025-03-17T16:45:06.723ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 128.656055ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2602025-03-17T16:45:06.725ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2612025-03-17T16:45:06.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2622025-03-17T16:45:06.727ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2632025-03-17T16:45:06.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2642025-03-17T16:45:06.728ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2652025-03-17T16:45:06.730ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
2662025-03-17T16:45:06.730ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2672025-03-17T16:45:06.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2682025-03-17T16:45:06.738ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2692025-03-17T16:45:06.743ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
2702025-03-17T16:45:06.743ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2712025-03-17T16:45:06.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2722025-03-17T16:45:06.745ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 150.998384ms iteration = 9 status = 0
2732025-03-17T16:45:06.749ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2742025-03-17T16:45:06.749ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2752025-03-17T16:45:06.749ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 154.844887ms iteration = 9 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2762025-03-17T16:45:06.751ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
2772025-03-17T16:45:06.751ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 156.925451ms iteration = 9 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
2782025-03-17T16:45:06.754ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2792025-03-17T16:45:06.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2802025-03-17T16:45:06.757ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 162.884188ms iteration = 9 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
2812025-03-17T16:45:06.757ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2822025-03-17T16:45:06.760ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
2832025-03-17T16:45:06.760ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2842025-03-17T16:45:06.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2852025-03-17T16:45:06.771ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
2862025-03-17T16:45:06.771ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 177.379409ms iteration = 5 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
2872025-03-17T16:45:06.796ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
2882025-03-17T16:45:06.796ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 202.271662ms iteration = 5 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
2892025-03-17T16:45:06.798ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2902025-03-17T16:45:06.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2912025-03-17T16:45:06.799ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2922025-03-17T16:45:06.801ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
2932025-03-17T16:45:06.801ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2942025-03-17T16:45:06.801ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2952025-03-17T16:45:06.801ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2962025-03-17T16:45:06.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2972025-03-17T16:45:06.802ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1758e36c-e0e6-4a0f-b193-16698a5c87e2", "content-length": "26", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
2982025-03-17T16:45:06.802ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2992025-03-17T16:45:06.802ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
3002025-03-17T16:45:06.802ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
3012025-03-17T16:45:06.802ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3022025-03-17T16:45:06.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3032025-03-17T16:45:06.804ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3042025-03-17T16:45:06.804ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
3052025-03-17T16:45:06.806ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3062025-03-17T16:45:06.806ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3072025-03-17T16:45:06.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3082025-03-17T16:45:06.813ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 218.492029ms iteration = 10 status = {"physical_disks_added":0}
3092025-03-17T16:45:06.849ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
3102025-03-17T16:45:06.849ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3112025-03-17T16:45:06.849ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "20e01b2d-d947-4124-bc80-2e4b5b724703", "content-length": "26", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3122025-03-17T16:45:06.849ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3132025-03-17T16:45:06.851ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3142025-03-17T16:45:06.853ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 259.409755ms iteration = 9 status = {}
3152025-03-17T16:45:06.858ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
3162025-03-17T16:45:06.858ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "1bec579d-ed53-40c0-84a4-5144c77cafb4", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3172025-03-17T16:45:06.858ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3182025-03-17T16:45:06.860ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "f599b15d-ee84-4672-b10f-858a73fae0e7", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3192025-03-17T16:45:06.860ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3202025-03-17T16:45:06.862ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3212025-03-17T16:45:06.862ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 268.596893ms iteration = 7 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
3222025-03-17T16:45:06.870ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
3232025-03-17T16:45:06.922ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
3242025-03-17T16:45:06.922ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3252025-03-17T16:45:06.922ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cb766eb-4f01-4b6b-b3a1-63ee6f15535c", "content-length": "372", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3262025-03-17T16:45:06.922ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3272025-03-17T16:45:06.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3282025-03-17T16:45:06.924ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3292025-03-17T16:45:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3302025-03-17T16:45:06.927ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3312025-03-17T16:45:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3322025-03-17T16:45:06.931ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
3332025-03-17T16:45:06.931ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
3342025-03-17T16:45:06.931ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
3352025-03-17T16:45:06.931ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 337.542389ms iteration = 5 status = {"generation":1}
3362025-03-17T16:45:06.944ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3372025-03-17T16:45:06.944ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ca0d11b-cf7c-4edf-b62d-8ae06d08daa3", "content-length": "1", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3382025-03-17T16:45:06.946ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3392025-03-17T16:45:06.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3402025-03-17T16:45:06.948ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 354.510102ms iteration = 9 status = null
3412025-03-17T16:45:06.953ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
3422025-03-17T16:45:06.954ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "97e03dd1-8386-41c3-8272-43e8dfcbe390", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
3432025-03-17T16:45:06.954ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
3442025-03-17T16:45:06.955ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b76bbf7a-5c95-482c-97e0-e242b2a4f8b7", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
3452025-03-17T16:45:06.955ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
3462025-03-17T16:45:06.957ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d078df7e-b3e1-4328-a3da-6211331aa20b", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
3472025-03-17T16:45:06.957ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
3482025-03-17T16:45:06.958ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
3492025-03-17T16:45:06.958ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 364.2168ms iteration = 7 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
3502025-03-17T16:45:06.975ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
3512025-03-17T16:45:06.975ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
3522025-03-17T16:45:06.975ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "366bf9f7-5359-482d-ae58-3fd2541a1428", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3532025-03-17T16:45:06.975ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3542025-03-17T16:45:06.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3552025-03-17T16:45:06.980ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3562025-03-17T16:45:06.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3572025-03-17T16:45:06.984ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3582025-03-17T16:45:06.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3592025-03-17T16:45:06.986ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
3602025-03-17T16:45:06.986ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3612025-03-17T16:45:06.987ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "dba14f0c-d5cb-45bf-a8f5-7fc644a84ef3", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3622025-03-17T16:45:06.989ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 394.610952ms iteration = 9 status = {}
3632025-03-17T16:45:07.022ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3642025-03-17T16:45:07.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3652025-03-17T16:45:07.024ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
3662025-03-17T16:45:07.024ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3672025-03-17T16:45:07.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3682025-03-17T16:45:07.035ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
3692025-03-17T16:45:07.035ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfc0a4a3-9745-45ea-937f-ee96dcc24763", "content-length": "134", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3702025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
3712025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
3722025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
3732025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
3742025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3752025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3762025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3772025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
3782025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3792025-03-17T16:45:07.035ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
3802025-03-17T16:45:07.036ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
3812025-03-17T16:45:07.036ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3822025-03-17T16:45:07.036ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
3832025-03-17T16:45:07.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3842025-03-17T16:45:07.036ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f7849b9-bf4d-4fc7-ac14-4ce140cd2361", "content-length": "691", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3852025-03-17T16:45:07.036ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
3862025-03-17T16:45:07.036ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
3872025-03-17T16:45:07.036ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
3882025-03-17T16:45:07.038ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 443.312094ms iteration = 9 status = {}
3892025-03-17T16:45:07.038ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
3902025-03-17T16:45:07.038ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc04e8d9-9aa0-4029-b146-ea39f5223c69", "content-length": "2", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3912025-03-17T16:45:07.038ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
3922025-03-17T16:45:07.038ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e3b52a3-ce1b-47fa-a04f-baaa931915e8", "content-length": "52", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3932025-03-17T16:45:07.038ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
3942025-03-17T16:45:07.038ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
3952025-03-17T16:45:07.130ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "16c1a80e-8bd8-41b7-96de-699d12e3e749", "date": "Mon, 17 Mar 2025 16:45:06 GMT"} })
3962025-03-17T16:45:07.130ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
3972025-03-17T16:45:07.130ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
3982025-03-17T16:45:07.131ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "76312b93-59cd-4a54-b4e3-76e823b7b022", "content-length": "682", "date": "Mon, 17 Mar 2025 16:45:07 GMT"} })
3992025-03-17T16:45:07.131ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4002025-03-17T16:45:07.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4012025-03-17T16:45:07.135ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4022025-03-17T16:45:07.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4032025-03-17T16:45:07.139ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4042025-03-17T16:45:07.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4052025-03-17T16:45:07.141ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4062025-03-17T16:45:07.141ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4072025-03-17T16:45:07.148ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 554.298751ms iteration = 9 status = {}
4082025-03-17T16:45:10.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 1b07967a-1fb7-4f48-8398-d7040fd89720 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
4092025-03-17T16:45:16.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
4102025-03-17T16:45:16.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4112025-03-17T16:45:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4122025-03-17T16:45:16.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
4132025-03-17T16:45:16.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4142025-03-17T16:45:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4152025-03-17T16:45:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4162025-03-17T16:45:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4172025-03-17T16:45:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4182025-03-17T16:45:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4192025-03-17T16:45:16.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4202025-03-17T16:45:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4212025-03-17T16:45:16.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4222025-03-17T16:45:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
4232025-03-17T16:45:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4242025-03-17T16:45:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4252025-03-17T16:45:16.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
4262025-03-17T16:45:16.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.247015ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
4272025-03-17T16:45:16.629ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4282025-03-17T16:45:16.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4292025-03-17T16:45:16.631ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.802438ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4302025-03-17T16:45:20.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 2cb17a2c-4fd7-4879-b1d5-2407edc1775a roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
4312025-03-17T16:45:20.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 2cb17a2c-4fd7-4879-b1d5-2407edc1775a resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
4322025-03-17T16:45:20.128ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3090 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 2cb17a2c-4fd7-4879-b1d5-2407edc1775a response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
4332025-03-17T16:45:20.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 2c425265-7dbe-4c04-936f-5eeaaf9b63ea roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
4342025-03-17T16:45:20.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 2c425265-7dbe-4c04-936f-5eeaaf9b63ea resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
4352025-03-17T16:45:20.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 1988 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 2c425265-7dbe-4c04-936f-5eeaaf9b63ea response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
4362025-03-17T16:45:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 231 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 1f2eee0a-f864-4deb-a1df-348e9220f409 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
4372025-03-17T16:45:26.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
4382025-03-17T16:45:26.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4392025-03-17T16:45:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4402025-03-17T16:45:26.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
4412025-03-17T16:45:26.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4422025-03-17T16:45:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4432025-03-17T16:45:26.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4442025-03-17T16:45:26.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4452025-03-17T16:45:26.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4462025-03-17T16:45:26.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.651338ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4472025-03-17T16:45:26.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4482025-03-17T16:45:26.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4492025-03-17T16:45:26.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4502025-03-17T16:45:26.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4512025-03-17T16:45:26.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4522025-03-17T16:45:26.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4532025-03-17T16:45:26.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
4542025-03-17T16:45:26.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4552025-03-17T16:45:26.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4562025-03-17T16:45:26.612ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
4572025-03-17T16:45:26.612ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.374434ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
4582025-03-17T16:45:30.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 485 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 1acf3562-302c-4a6b-8c85-eb9121cd8480 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
4592025-03-17T16:45:36.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
4602025-03-17T16:45:36.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
4612025-03-17T16:45:36.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
4622025-03-17T16:45:36.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
4632025-03-17T16:45:36.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4642025-03-17T16:45:36.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
4652025-03-17T16:45:36.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4662025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
4672025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4682025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4692025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
4702025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
4712025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4722025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
4732025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4742025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
4752025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
4762025-03-17T16:45:36.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
4772025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4782025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
4792025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4802025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4812025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
4822025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4832025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
4842025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4852025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
4862025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4872025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
4882025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4892025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
4902025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4912025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
4922025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4932025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
4942025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
4952025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
4962025-03-17T16:45:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4972025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4982025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4992025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5002025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5012025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5022025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5032025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5042025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5052025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5062025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5072025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5082025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5092025-03-17T16:45:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
5102025-03-17T16:45:36.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5112025-03-17T16:45:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4023, tv_nsec: 859509259 } })
5122025-03-17T16:45:36.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.379544ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
5132025-03-17T16:45:36.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
5142025-03-17T16:45:36.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4023, tv_nsec: 860242595 } })
5152025-03-17T16:45:36.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
5162025-03-17T16:45:36.599ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5172025-03-17T16:45:36.599ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5182025-03-17T16:45:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5192025-03-17T16:45:36.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5202025-03-17T16:45:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5212025-03-17T16:45:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5222025-03-17T16:45:36.602ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5232025-03-17T16:45:36.602ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5242025-03-17T16:45:36.602ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5252025-03-17T16:45:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5262025-03-17T16:45:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5272025-03-17T16:45:36.604ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5282025-03-17T16:45:36.604ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5292025-03-17T16:45:36.604ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5302025-03-17T16:45:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5312025-03-17T16:45:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
5322025-03-17T16:45:36.606ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5332025-03-17T16:45:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5342025-03-17T16:45:36.606ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5352025-03-17T16:45:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5362025-03-17T16:45:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5372025-03-17T16:45:36.608ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5382025-03-17T16:45:36.608ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5392025-03-17T16:45:36.609ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 14.357327ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
5402025-03-17T16:45:36.610ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5412025-03-17T16:45:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5422025-03-17T16:45:36.610ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5432025-03-17T16:45:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5442025-03-17T16:45:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5452025-03-17T16:45:36.611ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5462025-03-17T16:45:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5472025-03-17T16:45:36.612ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5482025-03-17T16:45:36.612ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5492025-03-17T16:45:36.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5502025-03-17T16:45:36.613ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 19.098459ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
5512025-03-17T16:45:36.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5522025-03-17T16:45:36.615ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5532025-03-17T16:45:36.615ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5542025-03-17T16:45:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5552025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5562025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5572025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5582025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
5592025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
5602025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
5612025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
5622025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
5632025-03-17T16:45:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
5642025-03-17T16:45:36.617ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5652025-03-17T16:45:36.618ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
5662025-03-17T16:45:36.618ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 23.598615ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
5672025-03-17T16:45:36.618ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5682025-03-17T16:45:36.618ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5692025-03-17T16:45:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5702025-03-17T16:45:36.620ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
5712025-03-17T16:45:36.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5722025-03-17T16:45:36.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5732025-03-17T16:45:36.620ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5742025-03-17T16:45:36.620ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5752025-03-17T16:45:36.621ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4023, tv_nsec: 882875280 } })
5762025-03-17T16:45:36.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5772025-03-17T16:45:36.621ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
5782025-03-17T16:45:36.622ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5792025-03-17T16:45:36.623ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5802025-03-17T16:45:36.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5812025-03-17T16:45:36.624ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 7.842488ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
5822025-03-17T16:45:36.631ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5832025-03-17T16:45:36.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5842025-03-17T16:45:36.634ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
5852025-03-17T16:45:36.634ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 40.704912ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
5862025-03-17T16:45:36.636ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5872025-03-17T16:45:36.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5882025-03-17T16:45:36.637ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5892025-03-17T16:45:36.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5902025-03-17T16:45:36.639ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5912025-03-17T16:45:36.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5922025-03-17T16:45:36.642ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5932025-03-17T16:45:36.642ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
5942025-03-17T16:45:36.642ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5952025-03-17T16:45:36.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5962025-03-17T16:45:36.652ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
5972025-03-17T16:45:36.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5982025-03-17T16:45:36.654ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
5992025-03-17T16:45:36.654ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6002025-03-17T16:45:36.654ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "968c23a8-5f59-4d0f-9593-579333a1ab37", "content-length": "26", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6012025-03-17T16:45:36.654ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6022025-03-17T16:45:36.655ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6032025-03-17T16:45:36.656ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 40.548439ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
6042025-03-17T16:45:36.661ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
6052025-03-17T16:45:36.661ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6062025-03-17T16:45:36.661ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "726d95e4-52a1-4d8e-8b1a-dd6e363efcd7", "content-length": "26", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6072025-03-17T16:45:36.661ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6082025-03-17T16:45:36.663ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6092025-03-17T16:45:36.668ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
6102025-03-17T16:45:36.668ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 73.876202ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
6112025-03-17T16:45:36.669ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 74.564807ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6122025-03-17T16:45:36.671ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6132025-03-17T16:45:36.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6142025-03-17T16:45:36.672ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 77.49541ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6152025-03-17T16:45:36.672ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6162025-03-17T16:45:36.675ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6172025-03-17T16:45:36.675ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6182025-03-17T16:45:36.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6192025-03-17T16:45:36.677ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6202025-03-17T16:45:36.681ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6212025-03-17T16:45:36.681ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6222025-03-17T16:45:36.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6232025-03-17T16:45:36.684ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 89.417025ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6242025-03-17T16:45:36.686ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6252025-03-17T16:45:36.686ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
6262025-03-17T16:45:36.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6272025-03-17T16:45:36.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6282025-03-17T16:45:36.721ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6292025-03-17T16:45:36.721ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
6302025-03-17T16:45:36.721ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 126.931779ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
6312025-03-17T16:45:36.727ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 132.783694ms iteration = 11 status = {"physical_disks_added":0}
6322025-03-17T16:45:36.747ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
6332025-03-17T16:45:36.747ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6342025-03-17T16:45:36.747ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "53f65f67-108a-4c12-8c6d-c5d7c67d767a", "content-length": "26", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6352025-03-17T16:45:36.747ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6362025-03-17T16:45:36.748ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
6372025-03-17T16:45:36.770ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 177.495672ms iteration = 10 status = {}
6382025-03-17T16:45:36.816ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6392025-03-17T16:45:36.816ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "a79b6b61-b60b-44d0-a03f-c210daf68647", "content-length": "1", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6402025-03-17T16:45:36.817ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6412025-03-17T16:45:36.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6422025-03-17T16:45:36.841ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6432025-03-17T16:45:36.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6442025-03-17T16:45:36.940ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
6452025-03-17T16:45:36.940ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
6462025-03-17T16:45:36.940ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6472025-03-17T16:45:36.940ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eddb7a2-374a-429d-b7a0-7604a19c251e", "content-length": "691", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6482025-03-17T16:45:36.940ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
6492025-03-17T16:45:36.940ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
6502025-03-17T16:45:36.940ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
6512025-03-17T16:45:36.940ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c06dc3-66d5-430a-93da-28933d2f891b", "content-length": "372", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6522025-03-17T16:45:36.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6532025-03-17T16:45:36.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6542025-03-17T16:45:36.941ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 346.497421ms iteration = 10 status = {}
6552025-03-17T16:45:36.942ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 348.814481ms iteration = 10 status = null
6562025-03-17T16:45:36.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6572025-03-17T16:45:36.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6582025-03-17T16:45:36.944ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6592025-03-17T16:45:36.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
6602025-03-17T16:45:36.947ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6612025-03-17T16:45:36.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6622025-03-17T16:45:36.949ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 354.959172ms iteration = 10 status = 0
6632025-03-17T16:45:36.979ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6642025-03-17T16:45:36.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6652025-03-17T16:45:36.981ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
6662025-03-17T16:45:36.981ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6672025-03-17T16:45:36.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6682025-03-17T16:45:36.983ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
6692025-03-17T16:45:36.983ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
6702025-03-17T16:45:36.983ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "84656bdc-0544-4c61-a3f4-c8c09c75b94d", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6712025-03-17T16:45:36.983ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6722025-03-17T16:45:36.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6732025-03-17T16:45:36.986ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6742025-03-17T16:45:36.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6752025-03-17T16:45:36.989ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6762025-03-17T16:45:36.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6772025-03-17T16:45:36.991ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
6782025-03-17T16:45:36.991ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7f6b3b9-7940-4f5f-92b0-de1216c3abc8", "content-length": "134", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6792025-03-17T16:45:36.991ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
6802025-03-17T16:45:36.991ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
6812025-03-17T16:45:36.991ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
6822025-03-17T16:45:36.991ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6832025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
6842025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
6852025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6862025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6872025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6882025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6892025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6902025-03-17T16:45:36.992ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6912025-03-17T16:45:36.992ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "06280b6b-061b-46d2-9bda-3e1e5931fec7", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6922025-03-17T16:45:36.993ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 399.112888ms iteration = 10 status = {}
6932025-03-17T16:45:36.993ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
6942025-03-17T16:45:36.993ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6952025-03-17T16:45:36.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6962025-03-17T16:45:36.995ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
6972025-03-17T16:45:36.995ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "8613e7a6-6c3b-40f9-8193-67fab008ec2d", "content-length": "2", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
6982025-03-17T16:45:36.995ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
6992025-03-17T16:45:36.995ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "04838288-ec07-4096-b477-cabd864a5c80", "content-length": "52", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
7002025-03-17T16:45:36.996ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
7012025-03-17T16:45:36.996ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
7022025-03-17T16:45:37.091ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0896985b-b934-4eda-be8e-9cb29fb2d4f3", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
7032025-03-17T16:45:37.091ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
7042025-03-17T16:45:37.091ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
7052025-03-17T16:45:37.092ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0e544c3-55c1-4e7c-8147-0f33b1ececb7", "content-length": "682", "date": "Mon, 17 Mar 2025 16:45:36 GMT"} })
7062025-03-17T16:45:37.092ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7072025-03-17T16:45:37.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7082025-03-17T16:45:37.096ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7092025-03-17T16:45:37.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7102025-03-17T16:45:37.100ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7112025-03-17T16:45:37.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7122025-03-17T16:45:37.102ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7132025-03-17T16:45:37.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7142025-03-17T16:45:37.109ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 514.415467ms iteration = 10 status = {}
7152025-03-17T16:45:40.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = fdf5cfe2-e9fe-48fa-be95-66c1b6d9b7e8 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
7162025-03-17T16:45:46.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
7172025-03-17T16:45:46.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7182025-03-17T16:45:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7192025-03-17T16:45:46.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
7202025-03-17T16:45:46.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7212025-03-17T16:45:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7222025-03-17T16:45:46.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7232025-03-17T16:45:46.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7242025-03-17T16:45:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7252025-03-17T16:45:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7262025-03-17T16:45:46.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7272025-03-17T16:45:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7282025-03-17T16:45:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7292025-03-17T16:45:46.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7302025-03-17T16:45:46.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7312025-03-17T16:45:46.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
7322025-03-17T16:45:46.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7332025-03-17T16:45:46.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7342025-03-17T16:45:46.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
7352025-03-17T16:45:46.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.552131ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
7362025-03-17T16:45:46.632ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.779369ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7372025-03-17T16:45:50.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = c6bb6da0-baeb-4336-b42b-87cee52121e1 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
7382025-03-17T16:45:50.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = c6bb6da0-baeb-4336-b42b-87cee52121e1 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
7392025-03-17T16:45:50.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2924 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = c6bb6da0-baeb-4336-b42b-87cee52121e1 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
7402025-03-17T16:45:50.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 68890480-7875-4117-b6e7-cd407a82116b roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
7412025-03-17T16:45:50.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 68890480-7875-4117-b6e7-cd407a82116b resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
7422025-03-17T16:45:50.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2601 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 68890480-7875-4117-b6e7-cd407a82116b response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
7432025-03-17T16:45:50.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 385 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 55db049e-0c36-46ab-9c2b-add1ba17dfd0 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
7442025-03-17T16:45:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
7452025-03-17T16:45:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7462025-03-17T16:45:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7472025-03-17T16:45:56.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
7482025-03-17T16:45:56.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7492025-03-17T16:45:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7502025-03-17T16:45:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7512025-03-17T16:45:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7522025-03-17T16:45:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7532025-03-17T16:45:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7542025-03-17T16:45:56.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7552025-03-17T16:45:56.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7562025-03-17T16:45:56.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7572025-03-17T16:45:56.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
7582025-03-17T16:45:56.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7592025-03-17T16:45:56.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7602025-03-17T16:45:56.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
7612025-03-17T16:45:56.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.876939ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
7622025-03-17T16:45:56.632ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7632025-03-17T16:45:56.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7642025-03-17T16:45:56.661ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.512286ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7652025-03-17T16:46:00.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 468 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = b28618b3-0c8c-44a2-89cd-2a01dad7b9fd response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
7662025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
7672025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
7682025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7692025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
7702025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
7712025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7722025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
7732025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
7742025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7752025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
7762025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7772025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
7782025-03-17T16:46:06.594ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:36:06.594229270 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
7792025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
7802025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
7812025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
7822025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
7832025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7842025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
7852025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7862025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
7872025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
7882025-03-17T16:46:06.594ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
7892025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
7902025-03-17T16:46:06.594ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7912025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7922025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.121\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
7932025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
7942025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
7952025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
7962025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
7972025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
7982025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7992025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
8002025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8012025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
8022025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
8032025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
8042025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
8052025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8062025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
8072025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
8082025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
8092025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8102025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
8112025-03-17T16:46:06.594ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
8122025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8132025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
8142025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
8152025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8162025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
8172025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
8182025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
8192025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
8202025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8212025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8222025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
8232025-03-17T16:46:06.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
8242025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8252025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
8262025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8272025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
8282025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8292025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8302025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
8312025-03-17T16:46:06.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8322025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
8332025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8342025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8352025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
8362025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8372025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8382025-03-17T16:46:06.595ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4053, tv_nsec: 856831152 } })
8392025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
8402025-03-17T16:46:06.595ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4053, tv_nsec: 856884183 } })
8412025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8422025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8432025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8442025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8452025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
8462025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8472025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8482025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
8492025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8502025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
8512025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8522025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8532025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8542025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
8552025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8562025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8572025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8582025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8592025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
8602025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.42346ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
8612025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8622025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
8632025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
8642025-03-17T16:46:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.471202ms iteration = 6 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
8652025-03-17T16:46:06.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8662025-03-17T16:46:06.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4053, tv_nsec: 859386146 } })
8672025-03-17T16:46:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8682025-03-17T16:46:06.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8692025-03-17T16:46:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
8702025-03-17T16:46:06.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8712025-03-17T16:46:06.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8722025-03-17T16:46:06.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4053, tv_nsec: 860556331 } })
8732025-03-17T16:46:06.599ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
8742025-03-17T16:46:06.599ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.563328ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8752025-03-17T16:46:06.600ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:36:06.594229270 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
8762025-03-17T16:46:06.600ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.900186ms iteration = 6 status = {"expiration":"2025-03-17T16:36:06.594229270Z","pruned":[]}
8772025-03-17T16:46:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8782025-03-17T16:46:06.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8792025-03-17T16:46:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8802025-03-17T16:46:06.601ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8812025-03-17T16:46:06.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8822025-03-17T16:46:06.603ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8832025-03-17T16:46:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8842025-03-17T16:46:06.603ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8852025-03-17T16:46:06.603ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
8862025-03-17T16:46:06.603ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8872025-03-17T16:46:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8882025-03-17T16:46:06.604ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 9.452572ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
8892025-03-17T16:46:06.604ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8902025-03-17T16:46:06.604ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8912025-03-17T16:46:06.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8922025-03-17T16:46:06.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
8932025-03-17T16:46:06.606ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8942025-03-17T16:46:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8952025-03-17T16:46:06.608ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8962025-03-17T16:46:06.608ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8972025-03-17T16:46:06.610ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8982025-03-17T16:46:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8992025-03-17T16:46:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9002025-03-17T16:46:06.610ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9012025-03-17T16:46:06.610ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 15.918001ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
9022025-03-17T16:46:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9032025-03-17T16:46:06.610ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9042025-03-17T16:46:06.611ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9052025-03-17T16:46:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9062025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
9072025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
9082025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
9092025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
9102025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
9112025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
9122025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9132025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
9142025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9152025-03-17T16:46:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9162025-03-17T16:46:06.619ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9172025-03-17T16:46:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9182025-03-17T16:46:06.619ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9192025-03-17T16:46:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9202025-03-17T16:46:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9212025-03-17T16:46:06.620ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9222025-03-17T16:46:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9232025-03-17T16:46:06.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9242025-03-17T16:46:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9252025-03-17T16:46:06.621ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9262025-03-17T16:46:06.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9272025-03-17T16:46:06.637ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9282025-03-17T16:46:06.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9292025-03-17T16:46:06.651ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9302025-03-17T16:46:06.651ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9312025-03-17T16:46:06.651ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 56.895948ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9322025-03-17T16:46:06.652ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9332025-03-17T16:46:06.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9342025-03-17T16:46:06.655ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9352025-03-17T16:46:06.655ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9362025-03-17T16:46:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9372025-03-17T16:46:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9382025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9392025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9402025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9412025-03-17T16:46:06.659ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9422025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
9432025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
9442025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9452025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9462025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9472025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
9482025-03-17T16:46:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9492025-03-17T16:46:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
9502025-03-17T16:46:06.660ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
9512025-03-17T16:46:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
9522025-03-17T16:46:06.661ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9532025-03-17T16:46:06.662ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9542025-03-17T16:46:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9552025-03-17T16:46:06.662ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9562025-03-17T16:46:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9572025-03-17T16:46:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9582025-03-17T16:46:06.663ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9592025-03-17T16:46:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
9602025-03-17T16:46:06.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9612025-03-17T16:46:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9622025-03-17T16:46:06.665ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9632025-03-17T16:46:06.665ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9642025-03-17T16:46:06.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9652025-03-17T16:46:06.665ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 70.875538ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
9662025-03-17T16:46:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9672025-03-17T16:46:06.666ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9682025-03-17T16:46:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9692025-03-17T16:46:06.666ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9702025-03-17T16:46:06.666ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9712025-03-17T16:46:06.667ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4053, tv_nsec: 929185661 } })
9722025-03-17T16:46:06.667ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
9732025-03-17T16:46:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
9742025-03-17T16:46:06.669ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9752025-03-17T16:46:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9762025-03-17T16:46:06.671ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9772025-03-17T16:46:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
9782025-03-17T16:46:06.672ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9792025-03-17T16:46:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9802025-03-17T16:46:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
9812025-03-17T16:46:06.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9822025-03-17T16:46:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9832025-03-17T16:46:06.679ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9842025-03-17T16:46:06.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9852025-03-17T16:46:06.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9862025-03-17T16:46:06.680ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9872025-03-17T16:46:06.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9882025-03-17T16:46:06.681ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9892025-03-17T16:46:06.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9902025-03-17T16:46:06.683ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9912025-03-17T16:46:06.683ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9922025-03-17T16:46:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9932025-03-17T16:46:06.689ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 73.390612ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
9942025-03-17T16:46:06.696ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9952025-03-17T16:46:06.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9962025-03-17T16:46:06.697ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9972025-03-17T16:46:06.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9982025-03-17T16:46:06.698ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 82.503817ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
9992025-03-17T16:46:06.701ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10002025-03-17T16:46:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10012025-03-17T16:46:06.701ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
10022025-03-17T16:46:06.701ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
10032025-03-17T16:46:06.701ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
10042025-03-17T16:46:06.701ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10052025-03-17T16:46:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10062025-03-17T16:46:06.702ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
10072025-03-17T16:46:06.702ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 108.176617ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
10082025-03-17T16:46:06.703ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10092025-03-17T16:46:06.704ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10102025-03-17T16:46:06.708ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
10112025-03-17T16:46:06.708ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
10122025-03-17T16:46:06.708ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
10132025-03-17T16:46:06.708ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10142025-03-17T16:46:06.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10152025-03-17T16:46:06.709ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10162025-03-17T16:46:06.709ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10172025-03-17T16:46:06.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10182025-03-17T16:46:06.710ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10192025-03-17T16:46:06.710ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
10202025-03-17T16:46:06.710ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10212025-03-17T16:46:06.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10222025-03-17T16:46:06.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10232025-03-17T16:46:06.712ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10242025-03-17T16:46:06.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10252025-03-17T16:46:06.713ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 118.712242ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10262025-03-17T16:46:06.713ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 118.874016ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10272025-03-17T16:46:06.715ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 121.193626ms iteration = 11 status = 0
10282025-03-17T16:46:06.717ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10292025-03-17T16:46:06.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10302025-03-17T16:46:06.719ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10312025-03-17T16:46:06.720ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10322025-03-17T16:46:06.720ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
10332025-03-17T16:46:06.720ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 126.118151ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
10342025-03-17T16:46:06.722ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
10352025-03-17T16:46:06.722ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10362025-03-17T16:46:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10372025-03-17T16:46:06.723ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10382025-03-17T16:46:06.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10392025-03-17T16:46:06.724ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10402025-03-17T16:46:06.724ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10412025-03-17T16:46:06.724ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 129.948973ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10422025-03-17T16:46:06.725ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10432025-03-17T16:46:06.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10442025-03-17T16:46:06.725ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10452025-03-17T16:46:06.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10462025-03-17T16:46:06.728ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
10472025-03-17T16:46:06.729ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10482025-03-17T16:46:06.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10492025-03-17T16:46:06.730ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
10502025-03-17T16:46:06.731ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
10512025-03-17T16:46:06.731ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 136.35536ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
10522025-03-17T16:46:06.731ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10532025-03-17T16:46:06.733ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
10542025-03-17T16:46:06.733ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
10552025-03-17T16:46:06.733ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7874f6bf-4f1d-4227-b088-717ca93af27d", "content-length": "26", "date": "Mon, 17 Mar 2025 16:46:06 GMT"} })
10562025-03-17T16:46:06.733ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
10572025-03-17T16:46:06.733ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10582025-03-17T16:46:06.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10592025-03-17T16:46:06.735ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
10602025-03-17T16:46:06.735ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10612025-03-17T16:46:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10622025-03-17T16:46:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
10632025-03-17T16:46:06.735ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10642025-03-17T16:46:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10652025-03-17T16:46:06.737ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10662025-03-17T16:46:06.738ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10672025-03-17T16:46:06.738ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10682025-03-17T16:46:06.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10692025-03-17T16:46:06.739ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
10702025-03-17T16:46:06.739ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
10712025-03-17T16:46:06.739ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10722025-03-17T16:46:06.740ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
10732025-03-17T16:46:06.740ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10742025-03-17T16:46:06.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
10752025-03-17T16:46:06.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10762025-03-17T16:46:06.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10772025-03-17T16:46:06.741ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10782025-03-17T16:46:06.742ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10792025-03-17T16:46:06.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10802025-03-17T16:46:06.744ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10812025-03-17T16:46:06.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10822025-03-17T16:46:06.746ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
10832025-03-17T16:46:06.746ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10842025-03-17T16:46:06.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10852025-03-17T16:46:06.752ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
10862025-03-17T16:46:06.752ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 157.8214ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
10872025-03-17T16:46:06.752ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
10882025-03-17T16:46:06.752ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
10892025-03-17T16:46:06.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10902025-03-17T16:46:06.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10912025-03-17T16:46:06.753ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10922025-03-17T16:46:06.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
10932025-03-17T16:46:06.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10942025-03-17T16:46:06.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10952025-03-17T16:46:06.757ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10962025-03-17T16:46:06.761ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10972025-03-17T16:46:06.761ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10982025-03-17T16:46:06.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10992025-03-17T16:46:06.762ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
11002025-03-17T16:46:06.762ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
11012025-03-17T16:46:06.762ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11022025-03-17T16:46:06.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11032025-03-17T16:46:06.763ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
11042025-03-17T16:46:06.763ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
11052025-03-17T16:46:06.763ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
11062025-03-17T16:46:06.763ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 169.52824ms iteration = 6 status = {"generation":2}
11072025-03-17T16:46:06.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11082025-03-17T16:46:06.764ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 169.864928ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11092025-03-17T16:46:06.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11102025-03-17T16:46:06.767ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11112025-03-17T16:46:06.767ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11122025-03-17T16:46:06.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11132025-03-17T16:46:06.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11142025-03-17T16:46:06.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11152025-03-17T16:46:06.770ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
11162025-03-17T16:46:06.770ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
11172025-03-17T16:46:06.770ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
11182025-03-17T16:46:06.770ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 175.921838ms iteration = 6 status = {"generation":1}
11192025-03-17T16:46:06.770ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
11202025-03-17T16:46:06.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11212025-03-17T16:46:06.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11222025-03-17T16:46:06.779ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
11232025-03-17T16:46:06.779ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 184.88372ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11242025-03-17T16:46:06.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11252025-03-17T16:46:06.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11262025-03-17T16:46:06.803ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
11272025-03-17T16:46:06.803ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 209.512318ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
11282025-03-17T16:46:06.854ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11292025-03-17T16:46:06.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11302025-03-17T16:46:06.856ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11312025-03-17T16:46:06.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11322025-03-17T16:46:06.857ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 262.362699ms iteration = 12 status = {"physical_disks_added":0}
11332025-03-17T16:46:06.858ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
11342025-03-17T16:46:06.858ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11352025-03-17T16:46:06.859ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "12e7c0b5-5c64-4617-a12f-ac95ccd1a2b2", "content-length": "26", "date": "Mon, 17 Mar 2025 16:46:06 GMT"} })
11362025-03-17T16:46:06.859ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11372025-03-17T16:46:06.860ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11382025-03-17T16:46:06.883ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
11392025-03-17T16:46:06.889ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
11402025-03-17T16:46:06.995ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
11412025-03-17T16:46:06.995ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11422025-03-17T16:46:06.995ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "107cac35-644b-4bbe-9e1b-a41742dbf2fd", "content-length": "26", "date": "Mon, 17 Mar 2025 16:46:06 GMT"} })
11432025-03-17T16:46:06.995ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11442025-03-17T16:46:06.997ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
11452025-03-17T16:46:06.997ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11462025-03-17T16:46:06.997ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cf9f1ab-33a7-4780-ac5e-371c19cdb83d", "content-length": "1", "date": "Mon, 17 Mar 2025 16:46:06 GMT"} })
11472025-03-17T16:46:06.999ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11482025-03-17T16:46:06.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11492025-03-17T16:46:07.047ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
11502025-03-17T16:46:07.047ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11512025-03-17T16:46:07.048ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d74904c-746a-403e-8d1d-fc36d14987f6", "content-length": "372", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
11522025-03-17T16:46:07.048ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11532025-03-17T16:46:07.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11542025-03-17T16:46:07.048ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11552025-03-17T16:46:07.050ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
11562025-03-17T16:46:07.050ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11572025-03-17T16:46:07.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11582025-03-17T16:46:07.059ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11592025-03-17T16:46:07.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11602025-03-17T16:46:07.061ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11612025-03-17T16:46:07.062ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11622025-03-17T16:46:07.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11632025-03-17T16:46:07.063ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 468.852052ms iteration = 11 status = null
11642025-03-17T16:46:07.063ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11652025-03-17T16:46:07.063ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11662025-03-17T16:46:07.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11672025-03-17T16:46:07.064ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11682025-03-17T16:46:07.064ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11692025-03-17T16:46:07.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11702025-03-17T16:46:07.066ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11712025-03-17T16:46:07.066ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11722025-03-17T16:46:07.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11732025-03-17T16:46:07.068ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11742025-03-17T16:46:07.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11752025-03-17T16:46:07.078ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
11762025-03-17T16:46:07.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
11772025-03-17T16:46:07.079ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
11782025-03-17T16:46:07.084ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "82d3a74f-f7c0-4c6f-a925-a94338e5c976", "content-length": "2", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
11792025-03-17T16:46:07.086ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
11802025-03-17T16:46:07.088ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 471.409246ms iteration = 2 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"8007234b-9b03-4ff0-af97-c65524f74b0a","time":"2025-03-17T16:46:07.086835970Z"}]}
11812025-03-17T16:46:07.098ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11822025-03-17T16:46:07.099ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11832025-03-17T16:46:07.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11842025-03-17T16:46:07.100ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11852025-03-17T16:46:07.100ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11862025-03-17T16:46:07.101ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11872025-03-17T16:46:07.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11882025-03-17T16:46:07.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11892025-03-17T16:46:07.103ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 508.93121ms iteration = 11 status = {}
11902025-03-17T16:46:07.103ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11912025-03-17T16:46:07.104ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
11922025-03-17T16:46:07.104ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
11932025-03-17T16:46:07.105ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8c332f3c-4ec3-4e28-8a93-836abaa37d82", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
11942025-03-17T16:46:07.105ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11952025-03-17T16:46:07.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11962025-03-17T16:46:07.107ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11972025-03-17T16:46:07.107ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11982025-03-17T16:46:07.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11992025-03-17T16:46:07.107ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12002025-03-17T16:46:07.107ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12012025-03-17T16:46:07.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12022025-03-17T16:46:07.108ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12032025-03-17T16:46:07.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12042025-03-17T16:46:07.109ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12052025-03-17T16:46:07.109ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12062025-03-17T16:46:07.109ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12072025-03-17T16:46:07.111ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12082025-03-17T16:46:07.111ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12092025-03-17T16:46:07.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12102025-03-17T16:46:07.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12112025-03-17T16:46:07.113ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12122025-03-17T16:46:07.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12132025-03-17T16:46:07.114ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
12142025-03-17T16:46:07.114ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
12152025-03-17T16:46:07.114ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "731d1ee7-9b69-4949-8378-d6716646b142", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
12162025-03-17T16:46:07.115ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12172025-03-17T16:46:07.115ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12182025-03-17T16:46:07.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12192025-03-17T16:46:07.116ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 521.368575ms iteration = 11 status = {}
12202025-03-17T16:46:07.116ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12212025-03-17T16:46:07.116ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12222025-03-17T16:46:07.116ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12232025-03-17T16:46:07.120ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12242025-03-17T16:46:07.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12252025-03-17T16:46:07.122ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12262025-03-17T16:46:07.123ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12272025-03-17T16:46:07.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12282025-03-17T16:46:07.123ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12292025-03-17T16:46:07.123ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12302025-03-17T16:46:07.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12312025-03-17T16:46:07.125ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12322025-03-17T16:46:07.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12332025-03-17T16:46:07.126ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12342025-03-17T16:46:07.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12352025-03-17T16:46:07.127ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12362025-03-17T16:46:07.127ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12372025-03-17T16:46:07.128ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12382025-03-17T16:46:07.129ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12392025-03-17T16:46:07.130ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12402025-03-17T16:46:07.130ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12412025-03-17T16:46:07.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12422025-03-17T16:46:07.135ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12432025-03-17T16:46:07.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12442025-03-17T16:46:07.135ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
12452025-03-17T16:46:07.135ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12462025-03-17T16:46:07.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12472025-03-17T16:46:07.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12482025-03-17T16:46:07.136ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12492025-03-17T16:46:07.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12502025-03-17T16:46:07.138ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12512025-03-17T16:46:07.138ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12522025-03-17T16:46:07.138ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12532025-03-17T16:46:07.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12542025-03-17T16:46:07.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12552025-03-17T16:46:07.139ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12562025-03-17T16:46:07.142ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12572025-03-17T16:46:07.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12582025-03-17T16:46:07.143ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
12592025-03-17T16:46:07.143ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12602025-03-17T16:46:07.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12612025-03-17T16:46:07.144ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12622025-03-17T16:46:07.145ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
12632025-03-17T16:46:07.146ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4fd43b65-c1bd-48ae-841e-9ec853ae0bc5", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12642025-03-17T16:46:07.146ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12652025-03-17T16:46:07.146ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12662025-03-17T16:46:07.146ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12672025-03-17T16:46:07.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12682025-03-17T16:46:07.147ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "417c344d-6705-4ae0-9687-9c7e2bd9db0a", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12692025-03-17T16:46:07.147ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12702025-03-17T16:46:07.148ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12712025-03-17T16:46:07.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12722025-03-17T16:46:07.148ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
12732025-03-17T16:46:07.149ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12742025-03-17T16:46:07.149ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 555.605269ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
12752025-03-17T16:46:07.150ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12762025-03-17T16:46:07.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12772025-03-17T16:46:07.150ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12782025-03-17T16:46:07.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12792025-03-17T16:46:07.151ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12802025-03-17T16:46:07.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12812025-03-17T16:46:07.152ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12822025-03-17T16:46:07.153ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12832025-03-17T16:46:07.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12842025-03-17T16:46:07.154ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12852025-03-17T16:46:07.154ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12862025-03-17T16:46:07.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12872025-03-17T16:46:07.155ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12882025-03-17T16:46:07.156ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12892025-03-17T16:46:07.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12902025-03-17T16:46:07.157ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12912025-03-17T16:46:07.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12922025-03-17T16:46:07.158ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
12932025-03-17T16:46:07.158ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
12942025-03-17T16:46:07.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
12952025-03-17T16:46:07.159ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12962025-03-17T16:46:07.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12972025-03-17T16:46:07.160ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12982025-03-17T16:46:07.163ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
12992025-03-17T16:46:07.165ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 568.209979ms iteration = 3 status = {"blueprint_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","inventory_collection_id":"d2c5a596-b855-4ff8-934a-9400c6884964","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":0,"num_already_tombstoned":0,"num_inserted":9,"num_not_in_inventory":0,"num_tombstoned":0}}}
13002025-03-17T16:46:07.165ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13012025-03-17T16:46:07.165ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13022025-03-17T16:46:07.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13032025-03-17T16:46:07.237ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
13042025-03-17T16:46:07.237ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "523390e4-1604-409d-992c-b97de63ea7fa", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13052025-03-17T16:46:07.237ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13062025-03-17T16:46:07.239ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "7fcf2f46-d9f4-47f8-99a5-3139948d03ba", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13072025-03-17T16:46:07.239ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13082025-03-17T16:46:07.241ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "6e4353f4-9282-4a5c-9c52-3cf16d784b65", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13092025-03-17T16:46:07.241ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13102025-03-17T16:46:07.243ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13112025-03-17T16:46:07.243ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 649.04728ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
13122025-03-17T16:46:07.247ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
13132025-03-17T16:46:07.247ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4be3bd86-6954-4750-b011-2bfc91f6caf7", "content-length": "691", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
13142025-03-17T16:46:07.247ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
13152025-03-17T16:46:07.247ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
13162025-03-17T16:46:07.247ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
13172025-03-17T16:46:07.248ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13182025-03-17T16:46:07.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13192025-03-17T16:46:07.249ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 654.227581ms iteration = 11 status = {}
13202025-03-17T16:46:07.250ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13212025-03-17T16:46:07.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13222025-03-17T16:46:07.251ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13232025-03-17T16:46:07.255ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13242025-03-17T16:46:07.255ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13252025-03-17T16:46:07.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13262025-03-17T16:46:07.261ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 4bbf4949-9681-44f4-9ce1-d34c7cfeaf8e, kind: Service { id: 3827727f-e72b-46db-972f-ef987798c7fe }, name: Name("nexus-3827727f-e72b-46db-972f-ef987798c7fe"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 212, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 64b2110d-a2ad-4bda-9407-3a44fd83a7ef, kind: Service { id: 807f84c2-ce3d-4190-8b1c-85a3b322601f }, name: Name("nexus-807f84c2-ce3d-4190-8b1c-85a3b322601f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c1e71a59-6914-4de0-a06a-4a40418bbcd0, kind: Service { id: b8f58801-e3d7-4a1a-b665-e14ca1b06822 }, name: Name("nexus-b8f58801-e3d7-4a1a-b665-e14ca1b06822"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 130])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 524f6c4e-dd14-4112-b941-145bff1dbf2b, kind: Service { id: 36bd5b8d-4179-4c7d-acfe-dc22937af04b }, name: Name("external-dns-36bd5b8d-4179-4c7d-acfe-dc22937af04b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 238])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 85143d5f-0adb-4664-a2ac-73adcf707df0, kind: Service { id: 91530a96-1dfe-4db2-b8a3-2a09e4675f90 }, name: Name("external-dns-91530a96-1dfe-4db2-b8a3-2a09e4675f90"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 26])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
13272025-03-17T16:46:07.262ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-03-17T16:46:07.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-03-17T16:46:07.263ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13302025-03-17T16:46:07.264ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13312025-03-17T16:46:07.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13322025-03-17T16:46:07.265ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
13332025-03-17T16:46:07.265ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13342025-03-17T16:46:07.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13352025-03-17T16:46:07.266ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
13362025-03-17T16:46:07.267ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13372025-03-17T16:46:07.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13382025-03-17T16:46:07.276ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
13392025-03-17T16:46:07.277ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8a523db-dd39-4c50-b243-f13444abd796", "content-length": "134", "date": "Mon, 17 Mar 2025 16:46:06 GMT"} })
13402025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
13412025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
13422025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
13432025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
13442025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13452025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13462025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13472025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13482025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
13492025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
13502025-03-17T16:46:07.277ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
13512025-03-17T16:46:07.277ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13522025-03-17T16:46:07.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13532025-03-17T16:46:07.279ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
13542025-03-17T16:46:07.279ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "262d5252-fbdb-4d6e-9614-52f6d5552644", "content-length": "2", "date": "Mon, 17 Mar 2025 16:46:06 GMT"} })
13552025-03-17T16:46:07.279ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
13562025-03-17T16:46:07.279ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "f61c7bc3-2455-4eb4-8efc-e65c8ab9f42f", "content-length": "52", "date": "Mon, 17 Mar 2025 16:46:06 GMT"} })
13572025-03-17T16:46:07.279ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
13582025-03-17T16:46:07.279ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
13592025-03-17T16:46:07.360ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
13602025-03-17T16:46:07.361ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 524f6c4e-dd14-4112-b941-145bff1dbf2b, kind: Service { id: 36bd5b8d-4179-4c7d-acfe-dc22937af04b }, name: Name("external-dns-36bd5b8d-4179-4c7d-acfe-dc22937af04b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 238])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 85143d5f-0adb-4664-a2ac-73adcf707df0, kind: Service { id: 91530a96-1dfe-4db2-b8a3-2a09e4675f90 }, name: Name("external-dns-91530a96-1dfe-4db2-b8a3-2a09e4675f90"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 26])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4bbf4949-9681-44f4-9ce1-d34c7cfeaf8e, kind: Service { id: 3827727f-e72b-46db-972f-ef987798c7fe }, name: Name("nexus-3827727f-e72b-46db-972f-ef987798c7fe"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 212, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 64b2110d-a2ad-4bda-9407-3a44fd83a7ef, kind: Service { id: 807f84c2-ce3d-4190-8b1c-85a3b322601f }, name: Name("nexus-807f84c2-ce3d-4190-8b1c-85a3b322601f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c1e71a59-6914-4de0-a06a-4a40418bbcd0, kind: Service { id: b8f58801-e3d7-4a1a-b665-e14ca1b06822 }, name: Name("nexus-b8f58801-e3d7-4a1a-b665-e14ca1b06822"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 130])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
13612025-03-17T16:46:07.361ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
13622025-03-17T16:46:07.386ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "c91876ec-8cd4-4c3f-bc4c-5886fa27c091", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
13632025-03-17T16:46:07.386ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
13642025-03-17T16:46:07.386ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
13652025-03-17T16:46:07.387ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d855fad5-e986-4109-93cc-551fe05338d1", "content-length": "682", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
13662025-03-17T16:46:07.387ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13672025-03-17T16:46:07.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13682025-03-17T16:46:07.399ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13692025-03-17T16:46:07.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13702025-03-17T16:46:07.415ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13712025-03-17T16:46:07.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13722025-03-17T16:46:07.430ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13732025-03-17T16:46:07.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13742025-03-17T16:46:07.444ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 849.263507ms iteration = 11 status = {}
13752025-03-17T16:46:07.625ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 8007234b-9b03-4ff0-af97-c65524f74b0a, time_created: 2025-03-17T16:41:47.612611Z, time_modified: 2025-03-17T16:41:47.612611Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
13762025-03-17T16:46:07.625ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
13772025-03-17T16:46:07.629ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13782025-03-17T16:46:07.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13792025-03-17T16:46:07.634ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13802025-03-17T16:46:07.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13812025-03-17T16:46:07.637ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13822025-03-17T16:46:07.643ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
13832025-03-17T16:46:07.677ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
13842025-03-17T16:46:07.678ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "cfe23145-8c89-4fa4-9b3c-7434f02e0d4f", "date": "Mon, 17 Mar 2025 16:46:07 GMT"} })
13852025-03-17T16:46:07.679ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
13862025-03-17T16:46:07.680ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 1.085556868s
13872025-03-17T16:46:07.680ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 1.085565509s iteration = 2 status = {}
13882025-03-17T16:46:10.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 308 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 60a4e7e0-6a0a-41f9-9afb-f40c1ca2d3eb response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
13892025-03-17T16:46:16.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
13902025-03-17T16:46:16.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13912025-03-17T16:46:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13922025-03-17T16:46:16.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
13932025-03-17T16:46:16.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13942025-03-17T16:46:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13952025-03-17T16:46:16.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13962025-03-17T16:46:16.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13972025-03-17T16:46:16.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13982025-03-17T16:46:16.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13992025-03-17T16:46:16.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14002025-03-17T16:46:16.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14012025-03-17T16:46:16.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14022025-03-17T16:46:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
14032025-03-17T16:46:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14042025-03-17T16:46:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14052025-03-17T16:46:16.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
14062025-03-17T16:46:16.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.421089ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
14072025-03-17T16:46:16.624ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14082025-03-17T16:46:16.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14092025-03-17T16:46:16.659ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.229507ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14102025-03-17T16:46:20.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 7ecb49d5-e789-4b74-a419-dda782a9deb1 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
14112025-03-17T16:46:20.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 7ecb49d5-e789-4b74-a419-dda782a9deb1 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
14122025-03-17T16:46:20.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2335 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 7ecb49d5-e789-4b74-a419-dda782a9deb1 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
14132025-03-17T16:46:20.127ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 11a1fd94-7652-4d63-870f-d8be6719d0de roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
14142025-03-17T16:46:20.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 11a1fd94-7652-4d63-870f-d8be6719d0de resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
14152025-03-17T16:46:20.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2286 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48706 req_id = 11a1fd94-7652-4d63-870f-d8be6719d0de response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
14162025-03-17T16:46:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 280 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = eca7e87d-491e-4452-a6ac-ab41240f828c response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
14172025-03-17T16:46:26.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
14182025-03-17T16:46:26.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14192025-03-17T16:46:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14202025-03-17T16:46:26.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
14212025-03-17T16:46:26.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14222025-03-17T16:46:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14232025-03-17T16:46:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14242025-03-17T16:46:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14252025-03-17T16:46:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14262025-03-17T16:46:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14272025-03-17T16:46:26.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14282025-03-17T16:46:26.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14292025-03-17T16:46:26.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14302025-03-17T16:46:26.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
14312025-03-17T16:46:26.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14322025-03-17T16:46:26.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14332025-03-17T16:46:26.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
14342025-03-17T16:46:26.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.314986ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
14352025-03-17T16:46:26.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14362025-03-17T16:46:26.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14372025-03-17T16:46:26.652ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.421072ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14382025-03-17T16:46:30.136ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 438 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 2ac9b2c0-989b-4ccb-b1ca-cb8f51ace382 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
14392025-03-17T16:46:36.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
14402025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
14412025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14422025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
14432025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14442025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
14452025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
14462025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14472025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14482025-03-17T16:46:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
14492025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
14502025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
14512025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
14522025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14532025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14542025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14552025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
14562025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14572025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
14582025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14592025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
14602025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
14612025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
14622025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
14632025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
14642025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14652025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
14662025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
14672025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14682025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
14692025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14702025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
14712025-03-17T16:46:36.595ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
14722025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
14732025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14742025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14752025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
14762025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
14772025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14782025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14792025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14802025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14812025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
14822025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14832025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
14842025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
14852025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14862025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
14872025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14882025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14892025-03-17T16:46:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14902025-03-17T16:46:36.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14912025-03-17T16:46:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4083, tv_nsec: 859191260 } })
14922025-03-17T16:46:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
14932025-03-17T16:46:36.598ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.23263ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
14942025-03-17T16:46:36.598ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14952025-03-17T16:46:36.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4083, tv_nsec: 860542928 } })
14962025-03-17T16:46:36.598ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14972025-03-17T16:46:36.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
14982025-03-17T16:46:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14992025-03-17T16:46:36.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15002025-03-17T16:46:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15012025-03-17T16:46:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15022025-03-17T16:46:36.601ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15032025-03-17T16:46:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15042025-03-17T16:46:36.602ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15052025-03-17T16:46:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
15062025-03-17T16:46:36.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15072025-03-17T16:46:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15082025-03-17T16:46:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
15092025-03-17T16:46:36.604ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15102025-03-17T16:46:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15112025-03-17T16:46:36.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15122025-03-17T16:46:36.607ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15132025-03-17T16:46:36.607ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15142025-03-17T16:46:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15152025-03-17T16:46:36.609ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15162025-03-17T16:46:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15172025-03-17T16:46:36.610ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
15182025-03-17T16:46:36.610ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15192025-03-17T16:46:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
15202025-03-17T16:46:36.610ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15212025-03-17T16:46:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15222025-03-17T16:46:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15232025-03-17T16:46:36.611ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15242025-03-17T16:46:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15252025-03-17T16:46:36.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15262025-03-17T16:46:36.614ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4083, tv_nsec: 876293776 } })
15272025-03-17T16:46:36.614ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
15282025-03-17T16:46:36.615ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
15292025-03-17T16:46:36.615ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15302025-03-17T16:46:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15312025-03-17T16:46:36.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
15322025-03-17T16:46:36.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
15332025-03-17T16:46:36.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
15342025-03-17T16:46:36.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
15352025-03-17T16:46:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
15362025-03-17T16:46:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
15372025-03-17T16:46:36.633ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15382025-03-17T16:46:36.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15392025-03-17T16:46:36.635ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15402025-03-17T16:46:36.636ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15412025-03-17T16:46:36.636ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15422025-03-17T16:46:36.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15432025-03-17T16:46:36.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15442025-03-17T16:46:36.637ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 42.524831ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
15452025-03-17T16:46:36.641ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
15462025-03-17T16:46:36.641ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15472025-03-17T16:46:36.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15482025-03-17T16:46:36.644ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15492025-03-17T16:46:36.644ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
15502025-03-17T16:46:36.644ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15512025-03-17T16:46:36.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
15522025-03-17T16:46:36.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15532025-03-17T16:46:36.645ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15542025-03-17T16:46:36.646ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
15552025-03-17T16:46:36.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
15562025-03-17T16:46:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
15572025-03-17T16:46:36.647ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15582025-03-17T16:46:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15592025-03-17T16:46:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15602025-03-17T16:46:36.649ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15612025-03-17T16:46:36.649ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 54.18594ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
15622025-03-17T16:46:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15632025-03-17T16:46:36.652ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15642025-03-17T16:46:36.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15652025-03-17T16:46:36.653ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
15662025-03-17T16:46:36.653ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 58.433971ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
15672025-03-17T16:46:36.656ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15682025-03-17T16:46:36.656ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
15692025-03-17T16:46:36.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15702025-03-17T16:46:36.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15712025-03-17T16:46:36.660ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 44.852251ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
15722025-03-17T16:46:36.664ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
15732025-03-17T16:46:36.664ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 70.336806ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
15742025-03-17T16:46:36.669ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 53.078047ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
15752025-03-17T16:46:36.675ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
15762025-03-17T16:46:36.675ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15772025-03-17T16:46:36.675ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3433d3b5-3d82-40e4-b488-21b1e13895d8", "content-length": "26", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
15782025-03-17T16:46:36.675ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15792025-03-17T16:46:36.677ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
15802025-03-17T16:46:36.677ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15812025-03-17T16:46:36.677ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15822025-03-17T16:46:36.677ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a4f75b1-0d5d-4427-a088-36d4e6d2fa1a", "content-length": "26", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
15832025-03-17T16:46:36.677ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15842025-03-17T16:46:36.679ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15852025-03-17T16:46:36.682ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15862025-03-17T16:46:36.682ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15872025-03-17T16:46:36.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15882025-03-17T16:46:36.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
15892025-03-17T16:46:36.685ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15902025-03-17T16:46:36.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15912025-03-17T16:46:36.686ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15922025-03-17T16:46:36.687ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
15932025-03-17T16:46:36.687ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
15942025-03-17T16:46:36.687ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "101aa56d-6108-48c3-b585-c2ce161bee34", "content-length": "26", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
15952025-03-17T16:46:36.687ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15962025-03-17T16:46:36.688ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 93.100524ms iteration = 12 status = 0
15972025-03-17T16:46:36.689ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
15982025-03-17T16:46:36.689ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
15992025-03-17T16:46:36.689ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16002025-03-17T16:46:36.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16012025-03-17T16:46:36.692ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 97.267203ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16022025-03-17T16:46:36.693ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16032025-03-17T16:46:36.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16042025-03-17T16:46:36.696ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 101.111636ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16052025-03-17T16:46:36.697ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16062025-03-17T16:46:36.697ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
16072025-03-17T16:46:36.697ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16082025-03-17T16:46:36.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16092025-03-17T16:46:36.726ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
16102025-03-17T16:46:36.726ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 131.630599ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
16112025-03-17T16:46:36.729ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 134.197564ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
16122025-03-17T16:46:36.735ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16132025-03-17T16:46:36.735ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
16142025-03-17T16:46:36.735ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 140.57708ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
16152025-03-17T16:46:36.738ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 144.25857ms iteration = 12 status = {}
16162025-03-17T16:46:36.742ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
16172025-03-17T16:46:36.743ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "55c92106-c31c-4edd-96a2-d56b9ad4c41e", "content-length": "1", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16182025-03-17T16:46:36.745ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16192025-03-17T16:46:36.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
16202025-03-17T16:46:36.747ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 153.097069ms iteration = 12 status = null
16212025-03-17T16:46:36.749ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 154.883107ms iteration = 13 status = {"physical_disks_added":0}
16222025-03-17T16:46:36.760ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
16232025-03-17T16:46:36.760ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
16242025-03-17T16:46:36.761ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddff370c-ed2d-4b4e-9698-120c6b252b06", "content-length": "372", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16252025-03-17T16:46:36.761ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16262025-03-17T16:46:36.761ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16272025-03-17T16:46:36.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16282025-03-17T16:46:36.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16292025-03-17T16:46:36.791ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16302025-03-17T16:46:36.793ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
16312025-03-17T16:46:36.793ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16322025-03-17T16:46:36.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16332025-03-17T16:46:36.794ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16342025-03-17T16:46:36.796ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
16352025-03-17T16:46:36.796ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16362025-03-17T16:46:36.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16372025-03-17T16:46:36.803ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
16382025-03-17T16:46:36.803ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
16392025-03-17T16:46:36.803ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "560584e4-8615-4356-8520-c1541fb064b3", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16402025-03-17T16:46:36.803ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16412025-03-17T16:46:36.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16422025-03-17T16:46:36.805ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16432025-03-17T16:46:36.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16442025-03-17T16:46:36.808ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16452025-03-17T16:46:36.809ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16462025-03-17T16:46:36.812ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
16472025-03-17T16:46:36.812ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
16482025-03-17T16:46:36.812ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0e0f187f-b9d0-4aa2-9a3f-8be7c9b955a8", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16492025-03-17T16:46:36.813ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16502025-03-17T16:46:36.813ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 218.707604ms iteration = 12 status = {}
16512025-03-17T16:46:36.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16522025-03-17T16:46:36.816ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
16532025-03-17T16:46:36.816ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16542025-03-17T16:46:36.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16552025-03-17T16:46:36.825ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
16562025-03-17T16:46:36.826ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc9feb60-5bd6-49ed-a91a-664e9efebe3b", "content-length": "134", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16572025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
16582025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
16592025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
16602025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
16612025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16622025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16632025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16642025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16652025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16662025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16672025-03-17T16:46:36.826ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
16682025-03-17T16:46:36.826ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16692025-03-17T16:46:36.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16702025-03-17T16:46:36.828ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
16712025-03-17T16:46:36.828ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2647e1c4-dc0c-424d-b406-1e0db5e19bbf", "content-length": "2", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16722025-03-17T16:46:36.828ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
16732025-03-17T16:46:36.829ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bd85a3-aa35-41f3-853b-89d3c34ee47c", "content-length": "52", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16742025-03-17T16:46:36.829ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
16752025-03-17T16:46:36.829ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
16762025-03-17T16:46:36.887ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
16772025-03-17T16:46:36.922ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8024517d-50ed-4c12-985b-592759bd42a2", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16782025-03-17T16:46:36.922ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
16792025-03-17T16:46:36.922ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
16802025-03-17T16:46:36.922ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8adf4dd0-d0a5-4345-b072-d20f342c87f2", "content-length": "691", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16812025-03-17T16:46:36.922ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
16822025-03-17T16:46:36.922ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
16832025-03-17T16:46:36.922ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
16842025-03-17T16:46:36.923ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d332db32-5294-4cf9-a111-9e7a9f2f566f", "content-length": "682", "date": "Mon, 17 Mar 2025 16:46:36 GMT"} })
16852025-03-17T16:46:36.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16862025-03-17T16:46:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16872025-03-17T16:46:36.923ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 328.638949ms iteration = 12 status = {}
16882025-03-17T16:46:36.926ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16892025-03-17T16:46:36.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16902025-03-17T16:46:36.929ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16912025-03-17T16:46:36.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16922025-03-17T16:46:36.932ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16932025-03-17T16:46:36.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16942025-03-17T16:46:36.939ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 344.071239ms iteration = 12 status = {}
16952025-03-17T16:46:40.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 274 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 7f71ce58-9f9e-4e3f-9fe1-038b43013a20 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
16962025-03-17T16:46:46.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
16972025-03-17T16:46:46.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16982025-03-17T16:46:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16992025-03-17T16:46:46.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
17002025-03-17T16:46:46.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17012025-03-17T16:46:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17022025-03-17T16:46:46.624ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17032025-03-17T16:46:46.624ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17042025-03-17T16:46:46.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17052025-03-17T16:46:46.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17062025-03-17T16:46:46.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17072025-03-17T16:46:46.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17082025-03-17T16:46:46.629ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.51804ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17092025-03-17T16:46:46.635ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17102025-03-17T16:46:46.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17112025-03-17T16:46:46.636ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17122025-03-17T16:46:46.639ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
17132025-03-17T16:46:46.639ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17142025-03-17T16:46:46.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17152025-03-17T16:46:46.646ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
17162025-03-17T16:46:46.646ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.492924ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
17172025-03-17T16:46:50.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 463 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 63789fb2-3951-4d01-a722-ee19b444dc8d response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
17182025-03-17T16:46:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
17192025-03-17T16:46:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17202025-03-17T16:46:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17212025-03-17T16:46:56.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
17222025-03-17T16:46:56.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17232025-03-17T16:46:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17242025-03-17T16:46:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17252025-03-17T16:46:56.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17262025-03-17T16:46:56.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17272025-03-17T16:46:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17282025-03-17T16:46:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17292025-03-17T16:46:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17302025-03-17T16:46:56.600ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 6.266824ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17312025-03-17T16:46:56.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17322025-03-17T16:46:56.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17332025-03-17T16:46:56.636ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17342025-03-17T16:46:56.639ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
17352025-03-17T16:46:56.639ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17362025-03-17T16:46:56.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17372025-03-17T16:46:56.649ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
17382025-03-17T16:46:56.649ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 55.942608ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
17392025-03-17T16:47:00.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = a77cd8ea-9006-48ba-a8b7-afaf8ec0da43 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
17402025-03-17T16:47:05.125ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:48606
17412025-03-17T16:47:05.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48606 req_id = 95f67e6f-3564-4989-b949-59fee9d1a139 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
17422025-03-17T16:47:05.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48606 req_id = 95f67e6f-3564-4989-b949-59fee9d1a139 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
17432025-03-17T16:47:05.128ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3533 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48606 req_id = 95f67e6f-3564-4989-b949-59fee9d1a139 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
17442025-03-17T16:47:05.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48606 req_id = 8a377c9e-aa75-4a5a-be13-f26ab0d0102e roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
17452025-03-17T16:47:05.129ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48606 req_id = 8a377c9e-aa75-4a5a-be13-f26ab0d0102e resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
17462025-03-17T16:47:05.131ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2468 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48606 req_id = 8a377c9e-aa75-4a5a-be13-f26ab0d0102e response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
17472025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
17482025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
17492025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
17502025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
17512025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17522025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
17532025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17542025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
17552025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
17562025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
17572025-03-17T16:47:06.593ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:37:06.593473892 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
17582025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
17592025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
17602025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
17612025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17622025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
17632025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
17642025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
17652025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
17662025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17672025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17682025-03-17T16:47:06.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
17692025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17702025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17712025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17722025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17732025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17742025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
17752025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
17762025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17772025-03-17T16:47:06.594ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
17782025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
17792025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 16\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
17802025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
17812025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
17822025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
17832025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
17842025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
17852025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
17862025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
17872025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
17882025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17892025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17902025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
17912025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17922025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
17932025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
17942025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17952025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
17962025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
17972025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
17982025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17992025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
18002025-03-17T16:47:06.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
18012025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
18022025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
18032025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18042025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
18052025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18062025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18072025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
18082025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
18092025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
18102025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
18112025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
18122025-03-17T16:47:06.594ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
18132025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
18142025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
18152025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
18162025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18172025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18182025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18192025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18202025-03-17T16:47:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4113, tv_nsec: 856482519 } })
18212025-03-17T16:47:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4113, tv_nsec: 856479379 } })
18222025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
18232025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18242025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
18252025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18262025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18272025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
18282025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18292025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18302025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18312025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18322025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18332025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
18342025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18352025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18362025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
18372025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
18382025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18392025-03-17T16:47:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18402025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
18412025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.595144ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
18422025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18432025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18442025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
18452025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.709377ms iteration = 7 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
18462025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18472025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18482025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18492025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18502025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18512025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18522025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18532025-03-17T16:47:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18542025-03-17T16:47:06.596ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18552025-03-17T16:47:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.119628ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
18562025-03-17T16:47:06.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4113, tv_nsec: 860580297 } })
18572025-03-17T16:47:06.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
18582025-03-17T16:47:06.598ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18592025-03-17T16:47:06.598ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18602025-03-17T16:47:06.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18612025-03-17T16:47:06.599ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:37:06.593473892 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
18622025-03-17T16:47:06.599ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.983768ms iteration = 7 status = {"expiration":"2025-03-17T16:37:06.593473892Z","pruned":[]}
18632025-03-17T16:47:06.600ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4113, tv_nsec: 862300284 } })
18642025-03-17T16:47:06.600ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
18652025-03-17T16:47:06.604ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 9.902032ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
18662025-03-17T16:47:06.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
18672025-03-17T16:47:06.617ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
18682025-03-17T16:47:06.617ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
18692025-03-17T16:47:06.617ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
18702025-03-17T16:47:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
18712025-03-17T16:47:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
18722025-03-17T16:47:06.649ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18732025-03-17T16:47:06.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18742025-03-17T16:47:06.650ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18752025-03-17T16:47:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18762025-03-17T16:47:06.651ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18772025-03-17T16:47:06.652ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18782025-03-17T16:47:06.652ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18792025-03-17T16:47:06.652ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18802025-03-17T16:47:06.653ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18812025-03-17T16:47:06.653ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
18822025-03-17T16:47:06.653ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
18832025-03-17T16:47:06.653ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 59.620706ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
18842025-03-17T16:47:06.655ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.081179ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
18852025-03-17T16:47:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
18862025-03-17T16:47:06.655ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18872025-03-17T16:47:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18882025-03-17T16:47:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18892025-03-17T16:47:06.656ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18902025-03-17T16:47:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18912025-03-17T16:47:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18922025-03-17T16:47:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
18932025-03-17T16:47:06.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
18942025-03-17T16:47:06.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18952025-03-17T16:47:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18962025-03-17T16:47:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18972025-03-17T16:47:06.657ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18982025-03-17T16:47:06.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18992025-03-17T16:47:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19002025-03-17T16:47:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19012025-03-17T16:47:06.659ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19022025-03-17T16:47:06.659ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19032025-03-17T16:47:06.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19042025-03-17T16:47:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19052025-03-17T16:47:06.660ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19062025-03-17T16:47:06.660ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
19072025-03-17T16:47:06.660ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19082025-03-17T16:47:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19092025-03-17T16:47:06.660ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19102025-03-17T16:47:06.660ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19112025-03-17T16:47:06.661ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19122025-03-17T16:47:06.662ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19132025-03-17T16:47:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19142025-03-17T16:47:06.662ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19152025-03-17T16:47:06.662ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19162025-03-17T16:47:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19172025-03-17T16:47:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19182025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
19192025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
19202025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
19212025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
19222025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19232025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19242025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
19252025-03-17T16:47:06.664ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
19262025-03-17T16:47:06.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
19272025-03-17T16:47:06.665ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19282025-03-17T16:47:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
19292025-03-17T16:47:06.666ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19302025-03-17T16:47:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19312025-03-17T16:47:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
19322025-03-17T16:47:06.667ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
19332025-03-17T16:47:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
19342025-03-17T16:47:06.667ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19352025-03-17T16:47:06.667ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19362025-03-17T16:47:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19372025-03-17T16:47:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
19382025-03-17T16:47:06.669ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
19392025-03-17T16:47:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
19402025-03-17T16:47:06.672ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19412025-03-17T16:47:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19422025-03-17T16:47:06.673ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19432025-03-17T16:47:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19442025-03-17T16:47:06.673ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19452025-03-17T16:47:06.675ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 81.540026ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19462025-03-17T16:47:06.676ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19472025-03-17T16:47:06.676ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19482025-03-17T16:47:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19492025-03-17T16:47:06.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4113, tv_nsec: 940669133 } })
19502025-03-17T16:47:06.678ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19512025-03-17T16:47:06.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
19522025-03-17T16:47:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19532025-03-17T16:47:06.679ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19542025-03-17T16:47:06.680ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19552025-03-17T16:47:06.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19562025-03-17T16:47:06.685ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19572025-03-17T16:47:06.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
19582025-03-17T16:47:06.685ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19592025-03-17T16:47:06.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19602025-03-17T16:47:06.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19612025-03-17T16:47:06.687ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19622025-03-17T16:47:06.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
19632025-03-17T16:47:06.691ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19642025-03-17T16:47:06.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19652025-03-17T16:47:06.703ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19662025-03-17T16:47:06.703ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19672025-03-17T16:47:06.703ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19682025-03-17T16:47:06.703ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19692025-03-17T16:47:06.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19702025-03-17T16:47:06.703ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
19712025-03-17T16:47:06.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
19722025-03-17T16:47:06.709ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732025-03-17T16:47:06.711ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19742025-03-17T16:47:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
19752025-03-17T16:47:06.711ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
19762025-03-17T16:47:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19772025-03-17T16:47:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
19782025-03-17T16:47:06.712ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 96.068218ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
19792025-03-17T16:47:06.713ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 119.393307ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19802025-03-17T16:47:06.714ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19812025-03-17T16:47:06.715ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19822025-03-17T16:47:06.715ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 121.665915ms iteration = 4 status = {"nsuccess":5}
19832025-03-17T16:47:06.716ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 122.415022ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19842025-03-17T16:47:06.717ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
19852025-03-17T16:47:06.717ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 123.656028ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
19862025-03-17T16:47:06.718ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 101.376721ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
19872025-03-17T16:47:06.719ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19882025-03-17T16:47:06.719ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19892025-03-17T16:47:06.719ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19902025-03-17T16:47:06.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19912025-03-17T16:47:06.721ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19922025-03-17T16:47:06.721ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
19932025-03-17T16:47:06.721ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19942025-03-17T16:47:06.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19952025-03-17T16:47:06.723ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
19962025-03-17T16:47:06.723ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19972025-03-17T16:47:06.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19982025-03-17T16:47:06.725ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19992025-03-17T16:47:06.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20002025-03-17T16:47:06.728ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
20012025-03-17T16:47:06.728ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
20022025-03-17T16:47:06.728ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 134.45034ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
20032025-03-17T16:47:06.731ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20042025-03-17T16:47:06.733ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
20052025-03-17T16:47:06.733ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 139.776813ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
20062025-03-17T16:47:06.734ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20072025-03-17T16:47:06.734ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
20082025-03-17T16:47:06.734ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20092025-03-17T16:47:06.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20102025-03-17T16:47:06.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20112025-03-17T16:47:06.736ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 141.890479ms iteration = 13 status = 0
20122025-03-17T16:47:06.736ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20132025-03-17T16:47:06.738ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
20142025-03-17T16:47:06.738ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20152025-03-17T16:47:06.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20162025-03-17T16:47:06.741ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20172025-03-17T16:47:06.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20182025-03-17T16:47:06.742ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
20192025-03-17T16:47:06.742ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20202025-03-17T16:47:06.743ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "28429646-655d-4c9e-8c90-01b162ed7b62", "content-length": "26", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
20212025-03-17T16:47:06.743ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20222025-03-17T16:47:06.743ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20232025-03-17T16:47:06.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20242025-03-17T16:47:06.744ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
20252025-03-17T16:47:06.758ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20262025-03-17T16:47:06.758ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
20272025-03-17T16:47:06.758ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 164.556135ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
20282025-03-17T16:47:06.758ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
20292025-03-17T16:47:06.758ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
20302025-03-17T16:47:06.758ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
20312025-03-17T16:47:06.758ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20322025-03-17T16:47:06.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20332025-03-17T16:47:06.760ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20342025-03-17T16:47:06.761ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 167.693842ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
20352025-03-17T16:47:06.763ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20362025-03-17T16:47:06.763ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20372025-03-17T16:47:06.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20382025-03-17T16:47:06.764ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20392025-03-17T16:47:06.764ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
20402025-03-17T16:47:06.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20412025-03-17T16:47:06.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20422025-03-17T16:47:06.765ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20432025-03-17T16:47:06.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20442025-03-17T16:47:06.768ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20452025-03-17T16:47:06.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20462025-03-17T16:47:06.771ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
20472025-03-17T16:47:06.771ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20482025-03-17T16:47:06.771ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
20492025-03-17T16:47:06.771ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 177.854209ms iteration = 7 status = {"generation":2}
20502025-03-17T16:47:06.775ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
20512025-03-17T16:47:06.775ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 181.499597ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
20522025-03-17T16:47:06.776ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
20532025-03-17T16:47:06.776ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 182.140821ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
20542025-03-17T16:47:06.789ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20552025-03-17T16:47:06.792ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20562025-03-17T16:47:06.792ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20572025-03-17T16:47:06.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20582025-03-17T16:47:06.803ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20592025-03-17T16:47:06.803ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20602025-03-17T16:47:06.803ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20612025-03-17T16:47:06.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20622025-03-17T16:47:06.805ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20632025-03-17T16:47:06.808ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20642025-03-17T16:47:06.808ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20652025-03-17T16:47:06.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20662025-03-17T16:47:06.812ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 218.022509ms iteration = 14 status = {"physical_disks_added":0}
20672025-03-17T16:47:06.846ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20682025-03-17T16:47:06.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20692025-03-17T16:47:06.861ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
20702025-03-17T16:47:06.907ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 314.333821ms iteration = 13 status = {}
20712025-03-17T16:47:06.907ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
20722025-03-17T16:47:06.917ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
20732025-03-17T16:47:06.917ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20742025-03-17T16:47:06.917ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e7b8c93-8148-44bc-8eb5-255df6a04d91", "content-length": "26", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
20752025-03-17T16:47:06.917ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20762025-03-17T16:47:06.919ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
20772025-03-17T16:47:06.938ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
20782025-03-17T16:47:06.938ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 345.260734ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
20792025-03-17T16:47:06.948ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
20802025-03-17T16:47:06.948ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20812025-03-17T16:47:06.949ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a39fea5f-635c-402e-8841-3cbd66c26734", "content-length": "26", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
20822025-03-17T16:47:06.949ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20832025-03-17T16:47:06.951ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
20842025-03-17T16:47:06.979ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
20852025-03-17T16:47:06.979ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20862025-03-17T16:47:06.979ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "da1e15be-e241-4a13-8401-23e764ebf758", "content-length": "372", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
20872025-03-17T16:47:06.979ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20882025-03-17T16:47:06.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20892025-03-17T16:47:06.982ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
20902025-03-17T16:47:06.982ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20912025-03-17T16:47:06.982ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
20922025-03-17T16:47:06.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20932025-03-17T16:47:06.982ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 388.802887ms iteration = 7 status = {"generation":1}
20942025-03-17T16:47:06.984ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20952025-03-17T16:47:06.984ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20962025-03-17T16:47:06.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20972025-03-17T16:47:07.014ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20982025-03-17T16:47:07.016ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20992025-03-17T16:47:07.016ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21002025-03-17T16:47:07.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21012025-03-17T16:47:07.025ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
21022025-03-17T16:47:07.025ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
21032025-03-17T16:47:07.025ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b7f34553-5ca1-43b8-b3d6-e81509039916", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21042025-03-17T16:47:07.025ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21052025-03-17T16:47:07.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21062025-03-17T16:47:07.027ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21072025-03-17T16:47:07.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21082025-03-17T16:47:07.030ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21092025-03-17T16:47:07.030ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21102025-03-17T16:47:07.033ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
21112025-03-17T16:47:07.033ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
21122025-03-17T16:47:07.033ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c99078d9-b5ce-4e76-bd70-ffb0e9e8df6f", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21132025-03-17T16:47:07.035ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 440.75582ms iteration = 13 status = {}
21142025-03-17T16:47:07.042ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
21152025-03-17T16:47:07.043ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "25027f12-5fba-46fa-bab6-987d7e0c06f3", "date": "Mon, 17 Mar 2025 16:47:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
21162025-03-17T16:47:07.043ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
21172025-03-17T16:47:07.044ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "4a2f0dfe-bd54-41d7-a247-bb1e111af6e3", "date": "Mon, 17 Mar 2025 16:47:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
21182025-03-17T16:47:07.044ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
21192025-03-17T16:47:07.046ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "be1d8a3a-ed87-40a8-8d0e-2e66af6a854e", "date": "Mon, 17 Mar 2025 16:47:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
21202025-03-17T16:47:07.046ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
21212025-03-17T16:47:07.048ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
21222025-03-17T16:47:07.048ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 454.590896ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
21232025-03-17T16:47:07.065ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
21242025-03-17T16:47:07.066ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3a594a9-3256-4e4c-b7da-e2bba4a7654a", "content-length": "1", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21252025-03-17T16:47:07.068ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21262025-03-17T16:47:07.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21272025-03-17T16:47:07.070ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 476.560427ms iteration = 13 status = null
21282025-03-17T16:47:07.121ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
21292025-03-17T16:47:07.121ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c529bdd0-d278-415d-b380-b24bc8300a2a", "date": "Mon, 17 Mar 2025 16:47:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21302025-03-17T16:47:07.121ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21312025-03-17T16:47:07.123ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "d244fec3-983f-43c3-b655-9d9f891c890b", "date": "Mon, 17 Mar 2025 16:47:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21322025-03-17T16:47:07.123ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21332025-03-17T16:47:07.124ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21342025-03-17T16:47:07.124ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 531.495774ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
21352025-03-17T16:47:07.133ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
21362025-03-17T16:47:07.134ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b6b7d75-e5cc-43c3-904f-fa2a65e8b4c8", "content-length": "691", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21372025-03-17T16:47:07.134ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
21382025-03-17T16:47:07.134ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
21392025-03-17T16:47:07.134ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
21402025-03-17T16:47:07.135ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 541.488717ms iteration = 13 status = {}
21412025-03-17T16:47:07.185ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21422025-03-17T16:47:07.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21432025-03-17T16:47:07.188ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
21442025-03-17T16:47:07.188ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21452025-03-17T16:47:07.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21462025-03-17T16:47:07.198ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
21472025-03-17T16:47:07.198ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "761f3176-303e-4982-9621-49b822f5bbbf", "content-length": "134", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21482025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
21492025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
21502025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
21512025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
21522025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21532025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21542025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21552025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
21562025-03-17T16:47:07.198ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
21572025-03-17T16:47:07.199ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
21582025-03-17T16:47:07.199ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
21592025-03-17T16:47:07.199ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21602025-03-17T16:47:07.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21612025-03-17T16:47:07.201ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
21622025-03-17T16:47:07.201ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0deaed9-bb92-4d9f-ac77-e3d9689cc6dc", "content-length": "2", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21632025-03-17T16:47:07.201ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
21642025-03-17T16:47:07.201ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "bad2a9f1-e059-4ad3-b383-37f1f2f9d688", "content-length": "52", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21652025-03-17T16:47:07.201ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
21662025-03-17T16:47:07.201ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
21672025-03-17T16:47:07.298ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8aca9f27-283d-4dae-b130-2ada7e7ae41a", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21682025-03-17T16:47:07.298ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
21692025-03-17T16:47:07.298ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
21702025-03-17T16:47:07.298ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9de8ca90-2026-42d7-ace6-3a180416809d", "content-length": "682", "date": "Mon, 17 Mar 2025 16:47:06 GMT"} })
21712025-03-17T16:47:07.298ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21722025-03-17T16:47:07.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21732025-03-17T16:47:07.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21742025-03-17T16:47:07.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21752025-03-17T16:47:07.306ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21762025-03-17T16:47:07.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21772025-03-17T16:47:07.313ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21782025-03-17T16:47:07.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21792025-03-17T16:47:07.320ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 726.54117ms iteration = 13 status = {}
21802025-03-17T16:47:07.466ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
21812025-03-17T16:47:07.466ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
21822025-03-17T16:47:07.510ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
21832025-03-17T16:47:07.510ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:35636
21842025-03-17T16:47:07.510ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35636 req_id = b7b81578-2b6a-418b-ac34-aa47ee93fb33 roles = RoleSet { roles: {} } uri = /metrics/producers
21852025-03-17T16:47:07.511ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35636 req_id = b7b81578-2b6a-418b-ac34-aa47ee93fb33 resource = Database result = Ok(()) uri = /metrics/producers
21862025-03-17T16:47:07.566ZINFOnexus: assigned collector to new producer collector_id = 1374f180-03c9-4b69-abf3-a0c92428bb9a file = nexus/src/app/oximeter.rs:76 producer_id = 807f84c2-ce3d-4190-8b1c-85a3b322601f
21872025-03-17T16:47:07.566ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 56000 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:35636 req_id = b7b81578-2b6a-418b-ac34-aa47ee93fb33 response_code = 201 uri = /metrics/producers
21882025-03-17T16:47:07.567ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "b7b81578-2b6a-418b-ac34-aa47ee93fb33", "content-length": "41", "date": "Mon, 17 Mar 2025 16:47:07 GMT"} })
21892025-03-17T16:47:07.569ZDEBGnexus (producer-registration-task): registered with nexus successfully
21902025-03-17T16:47:07.569ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
21912025-03-17T16:47:10.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = f71dee95-9cfd-41a8-aaac-19f2495c16fe response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
21922025-03-17T16:47:16.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
21932025-03-17T16:47:16.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21942025-03-17T16:47:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21952025-03-17T16:47:16.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
21962025-03-17T16:47:16.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21972025-03-17T16:47:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21982025-03-17T16:47:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21992025-03-17T16:47:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22002025-03-17T16:47:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22012025-03-17T16:47:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22022025-03-17T16:47:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22032025-03-17T16:47:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22042025-03-17T16:47:16.628ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22052025-03-17T16:47:16.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22062025-03-17T16:47:16.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22072025-03-17T16:47:16.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
22082025-03-17T16:47:16.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22092025-03-17T16:47:16.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22102025-03-17T16:47:16.641ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
22112025-03-17T16:47:16.641ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 48.499398ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
22122025-03-17T16:47:16.658ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.105783ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22132025-03-17T16:47:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 066299a1-0de5-4bc0-85a7-c25cbf3f348f response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
22142025-03-17T16:47:26.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
22152025-03-17T16:47:26.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22162025-03-17T16:47:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22172025-03-17T16:47:26.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
22182025-03-17T16:47:26.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22192025-03-17T16:47:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22202025-03-17T16:47:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22212025-03-17T16:47:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22222025-03-17T16:47:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22232025-03-17T16:47:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22242025-03-17T16:47:26.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22252025-03-17T16:47:26.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22262025-03-17T16:47:26.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22272025-03-17T16:47:26.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22282025-03-17T16:47:26.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22292025-03-17T16:47:26.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
22302025-03-17T16:47:26.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22312025-03-17T16:47:26.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22322025-03-17T16:47:26.640ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
22332025-03-17T16:47:26.640ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 47.586669ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
22342025-03-17T16:47:26.653ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.060305ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22352025-03-17T16:47:30.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 422 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 3d01d052-75ec-4069-a6ec-e3c9e546b42b response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
22362025-03-17T16:47:36.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
22372025-03-17T16:47:36.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
22382025-03-17T16:47:36.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22392025-03-17T16:47:36.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22402025-03-17T16:47:36.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
22412025-03-17T16:47:36.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
22422025-03-17T16:47:36.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
22432025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22442025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22452025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
22462025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
22472025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
22482025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22492025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
22502025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22512025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
22522025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
22532025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
22542025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
22552025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22562025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
22572025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
22582025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22592025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
22602025-03-17T16:47:36.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
22612025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22622025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
22632025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
22642025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22652025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
22662025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
22672025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22682025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
22692025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
22702025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
22712025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22722025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
22732025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22742025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22752025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22762025-03-17T16:47:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22772025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
22782025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22792025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22802025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22812025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22822025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
22832025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22842025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22852025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22862025-03-17T16:47:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22872025-03-17T16:47:36.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22882025-03-17T16:47:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4143, tv_nsec: 859162335 } })
22892025-03-17T16:47:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
22902025-03-17T16:47:36.598ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.590408ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
22912025-03-17T16:47:36.598ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22922025-03-17T16:47:36.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4143, tv_nsec: 860727308 } })
22932025-03-17T16:47:36.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
22942025-03-17T16:47:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22952025-03-17T16:47:36.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22962025-03-17T16:47:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22972025-03-17T16:47:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22982025-03-17T16:47:36.601ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22992025-03-17T16:47:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23002025-03-17T16:47:36.601ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23012025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23022025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23032025-03-17T16:47:36.603ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23042025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
23052025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23062025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23072025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
23082025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23092025-03-17T16:47:36.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23102025-03-17T16:47:36.605ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23112025-03-17T16:47:36.606ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23122025-03-17T16:47:36.606ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23132025-03-17T16:47:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
23142025-03-17T16:47:36.608ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23152025-03-17T16:47:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23162025-03-17T16:47:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23172025-03-17T16:47:36.609ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23182025-03-17T16:47:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23192025-03-17T16:47:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
23202025-03-17T16:47:36.609ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23212025-03-17T16:47:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23222025-03-17T16:47:36.611ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23232025-03-17T16:47:36.611ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 17.313841ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23242025-03-17T16:47:36.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23252025-03-17T16:47:36.612ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23262025-03-17T16:47:36.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23272025-03-17T16:47:36.613ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23282025-03-17T16:47:36.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23292025-03-17T16:47:36.615ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4143, tv_nsec: 877215181 } })
23302025-03-17T16:47:36.615ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
23312025-03-17T16:47:36.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
23322025-03-17T16:47:36.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
23332025-03-17T16:47:36.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
23342025-03-17T16:47:36.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
23352025-03-17T16:47:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
23362025-03-17T16:47:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
23372025-03-17T16:47:36.618ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
23382025-03-17T16:47:36.618ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 23.509103ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
23392025-03-17T16:47:36.634ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23402025-03-17T16:47:36.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23412025-03-17T16:47:36.642ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23422025-03-17T16:47:36.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23432025-03-17T16:47:36.645ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23442025-03-17T16:47:36.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23452025-03-17T16:47:36.646ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23462025-03-17T16:47:36.646ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
23472025-03-17T16:47:36.646ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23482025-03-17T16:47:36.647ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23492025-03-17T16:47:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23502025-03-17T16:47:36.647ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
23512025-03-17T16:47:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
23522025-03-17T16:47:36.648ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23532025-03-17T16:47:36.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23542025-03-17T16:47:36.648ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 53.905615ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
23552025-03-17T16:47:36.649ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23562025-03-17T16:47:36.649ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23572025-03-17T16:47:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23582025-03-17T16:47:36.650ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23592025-03-17T16:47:36.651ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 56.719734ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
23602025-03-17T16:47:36.653ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23612025-03-17T16:47:36.653ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23622025-03-17T16:47:36.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23632025-03-17T16:47:36.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23642025-03-17T16:47:36.659ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 43.544403ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
23652025-03-17T16:47:36.660ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23662025-03-17T16:47:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23672025-03-17T16:47:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
23682025-03-17T16:47:36.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23692025-03-17T16:47:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23702025-03-17T16:47:36.662ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23712025-03-17T16:47:36.662ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23722025-03-17T16:47:36.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23732025-03-17T16:47:36.665ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23742025-03-17T16:47:36.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23752025-03-17T16:47:36.665ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23762025-03-17T16:47:36.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23772025-03-17T16:47:36.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23782025-03-17T16:47:36.668ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 51.965953ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
23792025-03-17T16:47:36.670ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
23802025-03-17T16:47:36.670ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 76.470357ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
23812025-03-17T16:47:36.674ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23822025-03-17T16:47:36.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23832025-03-17T16:47:36.681ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 86.787429ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
23842025-03-17T16:47:36.688ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
23852025-03-17T16:47:36.688ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23862025-03-17T16:47:36.688ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "80ba311a-bb2f-4af3-9ec5-ec2bbccead22", "content-length": "26", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
23872025-03-17T16:47:36.688ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23882025-03-17T16:47:36.689ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23892025-03-17T16:47:36.689ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
23902025-03-17T16:47:36.689ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 94.751029ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
23912025-03-17T16:47:36.690ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23922025-03-17T16:47:36.691ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23932025-03-17T16:47:36.694ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
23942025-03-17T16:47:36.694ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 99.4365ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
23952025-03-17T16:47:36.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
23962025-03-17T16:47:36.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23972025-03-17T16:47:36.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23982025-03-17T16:47:36.696ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 101.689598ms iteration = 14 status = 0
23992025-03-17T16:47:36.698ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
24002025-03-17T16:47:36.698ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24012025-03-17T16:47:36.698ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce387596-1318-4332-980c-f14cdf06e8a8", "content-length": "26", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24022025-03-17T16:47:36.698ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24032025-03-17T16:47:36.700ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24042025-03-17T16:47:36.701ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
24052025-03-17T16:47:36.701ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24062025-03-17T16:47:36.701ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec3d7139-1068-42d3-a17f-dfe7662996e8", "content-length": "26", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24072025-03-17T16:47:36.702ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24082025-03-17T16:47:36.703ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
24092025-03-17T16:47:36.707ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 113.197495ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
24102025-03-17T16:47:36.732ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24112025-03-17T16:47:36.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24122025-03-17T16:47:36.760ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 165.655198ms iteration = 15 status = {"physical_disks_added":0}
24132025-03-17T16:47:36.819ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 225.782976ms iteration = 14 status = {}
24142025-03-17T16:47:36.829ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24152025-03-17T16:47:36.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24162025-03-17T16:47:36.829ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
24172025-03-17T16:47:36.829ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24182025-03-17T16:47:36.830ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "30306a31-4902-4873-ad0d-1a3b7a0e7a45", "content-length": "372", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24192025-03-17T16:47:36.830ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24202025-03-17T16:47:36.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24212025-03-17T16:47:36.845ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
24222025-03-17T16:47:36.845ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
24232025-03-17T16:47:36.845ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b9fcb024-6836-4b5e-9d3a-a1a116f23cd4", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24242025-03-17T16:47:36.845ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24252025-03-17T16:47:36.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24262025-03-17T16:47:36.848ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24272025-03-17T16:47:36.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24282025-03-17T16:47:36.851ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24292025-03-17T16:47:36.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24302025-03-17T16:47:36.854ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
24312025-03-17T16:47:36.854ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24322025-03-17T16:47:36.854ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c365dd74-7146-477e-ba9e-8d25a98a0957", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24332025-03-17T16:47:36.856ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 261.781886ms iteration = 14 status = {}
24342025-03-17T16:47:36.857ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
24352025-03-17T16:47:36.857ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "03aaa0e1-4247-4148-b7aa-a4d89458948e", "content-length": "1", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24362025-03-17T16:47:36.858ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24372025-03-17T16:47:36.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24382025-03-17T16:47:36.859ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24392025-03-17T16:47:36.860ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 267.162152ms iteration = 14 status = null
24402025-03-17T16:47:36.861ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24412025-03-17T16:47:36.861ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24422025-03-17T16:47:36.861ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24432025-03-17T16:47:36.863ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24442025-03-17T16:47:36.865ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24452025-03-17T16:47:36.865ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24462025-03-17T16:47:36.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24472025-03-17T16:47:36.993ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
24482025-03-17T16:47:36.993ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "be6722f7-5cd0-4d60-8f5e-33a54528ac38", "content-length": "691", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24492025-03-17T16:47:36.993ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
24502025-03-17T16:47:36.993ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
24512025-03-17T16:47:36.993ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
24522025-03-17T16:47:36.994ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 400.181381ms iteration = 14 status = {}
24532025-03-17T16:47:37.029ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24542025-03-17T16:47:37.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24552025-03-17T16:47:37.031ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
24562025-03-17T16:47:37.031ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24572025-03-17T16:47:37.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24582025-03-17T16:47:37.043ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
24592025-03-17T16:47:37.043ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9ef6c8f-b521-4942-91c1-126771465d52", "content-length": "134", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24602025-03-17T16:47:37.043ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
24612025-03-17T16:47:37.043ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
24622025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
24632025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
24642025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24652025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24662025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24672025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24682025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24692025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24702025-03-17T16:47:37.044ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
24712025-03-17T16:47:37.044ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24722025-03-17T16:47:37.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24732025-03-17T16:47:37.046ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
24742025-03-17T16:47:37.046ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "b39613fa-9d57-407d-b849-f2a25dc52972", "content-length": "2", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24752025-03-17T16:47:37.046ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
24762025-03-17T16:47:37.046ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "54b0a0c8-f874-474f-9e4f-4c735b034f6b", "content-length": "52", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24772025-03-17T16:47:37.046ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
24782025-03-17T16:47:37.046ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24792025-03-17T16:47:37.139ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "713e352d-c18c-4db9-9ae4-10b0aac86e2b", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24802025-03-17T16:47:37.139ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
24812025-03-17T16:47:37.139ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24822025-03-17T16:47:37.140ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab3aed09-73bb-462c-86f1-499390de35c4", "content-length": "682", "date": "Mon, 17 Mar 2025 16:47:36 GMT"} })
24832025-03-17T16:47:37.140ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24842025-03-17T16:47:37.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24852025-03-17T16:47:37.144ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24862025-03-17T16:47:37.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24872025-03-17T16:47:37.146ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24882025-03-17T16:47:37.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24892025-03-17T16:47:37.149ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24902025-03-17T16:47:37.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24912025-03-17T16:47:37.155ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 561.136098ms iteration = 14 status = {}
24922025-03-17T16:47:40.136ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 477 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 34c7a8e0-4ac7-44b9-9049-f95b114b9ad6 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
24932025-03-17T16:47:46.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
24942025-03-17T16:47:46.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24952025-03-17T16:47:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24962025-03-17T16:47:46.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
24972025-03-17T16:47:46.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24982025-03-17T16:47:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24992025-03-17T16:47:46.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25002025-03-17T16:47:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25012025-03-17T16:47:46.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25022025-03-17T16:47:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25032025-03-17T16:47:46.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25042025-03-17T16:47:46.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25052025-03-17T16:47:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25062025-03-17T16:47:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
25072025-03-17T16:47:46.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25082025-03-17T16:47:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25092025-03-17T16:47:46.612ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
25102025-03-17T16:47:46.612ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.270292ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
25112025-03-17T16:47:46.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25122025-03-17T16:47:46.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25132025-03-17T16:47:46.652ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 57.980531ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25142025-03-17T16:47:50.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 298 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 272805ce-e409-49d2-97cb-e02e085c4ae4 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
25152025-03-17T16:47:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
25162025-03-17T16:47:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25172025-03-17T16:47:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25182025-03-17T16:47:56.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
25192025-03-17T16:47:56.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25202025-03-17T16:47:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25212025-03-17T16:47:56.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25222025-03-17T16:47:56.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25232025-03-17T16:47:56.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25242025-03-17T16:47:56.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25252025-03-17T16:47:56.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25262025-03-17T16:47:56.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25272025-03-17T16:47:56.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25282025-03-17T16:47:56.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
25292025-03-17T16:47:56.607ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25302025-03-17T16:47:56.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25312025-03-17T16:47:56.618ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
25322025-03-17T16:47:56.618ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.471864ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
25332025-03-17T16:47:56.631ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25342025-03-17T16:47:56.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25352025-03-17T16:47:56.633ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.194299ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25362025-03-17T16:48:00.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 308 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = e7d1b9d0-736f-4cd9-b0ac-7d9b4a019c48 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
25372025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
25382025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
25392025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
25402025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25412025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
25422025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
25432025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25442025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
25452025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25462025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
25472025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
25482025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
25492025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25502025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25512025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
25522025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
25532025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
25542025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
25552025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
25562025-03-17T16:48:06.593ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:38:06.593415720 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
25572025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
25582025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
25592025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25602025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25612025-03-17T16:48:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25622025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25632025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25642025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
25652025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
25662025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
25672025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
25682025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
25692025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
25702025-03-17T16:48:06.594ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
25712025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
25722025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
25732025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 19.941\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
25742025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
25752025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
25762025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
25772025-03-17T16:48:06.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
25782025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25792025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25802025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
25812025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25822025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
25832025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25842025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
25852025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
25862025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
25872025-03-17T16:48:06.594ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
25882025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
25892025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
25902025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25912025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
25922025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
25932025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
25942025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
25952025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
25962025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25972025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25982025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
25992025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26002025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
26012025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26022025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
26032025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26042025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
26052025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26062025-03-17T16:48:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4173, tv_nsec: 856661009 } })
26072025-03-17T16:48:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4173, tv_nsec: 856664179 } })
26082025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26092025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
26102025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26112025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
26122025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
26132025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26142025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26152025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26162025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26172025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26182025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26192025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
26202025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26212025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26222025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
26232025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26242025-03-17T16:48:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
26252025-03-17T16:48:06.595ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
26262025-03-17T16:48:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
26272025-03-17T16:48:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.663986ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
26282025-03-17T16:48:06.595ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
26292025-03-17T16:48:06.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.812629ms iteration = 8 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
26302025-03-17T16:48:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26312025-03-17T16:48:06.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26322025-03-17T16:48:06.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26332025-03-17T16:48:06.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4173, tv_nsec: 859438238 } })
26342025-03-17T16:48:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
26352025-03-17T16:48:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.295532ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
26362025-03-17T16:48:06.598ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26372025-03-17T16:48:06.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26382025-03-17T16:48:06.599ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26392025-03-17T16:48:06.599ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:38:06.593415720 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
26402025-03-17T16:48:06.599ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.148021ms iteration = 8 status = {"expiration":"2025-03-17T16:38:06.593415720Z","pruned":[]}
26412025-03-17T16:48:06.599ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4173, tv_nsec: 862004553 } })
26422025-03-17T16:48:06.600ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
26432025-03-17T16:48:06.601ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
26442025-03-17T16:48:06.601ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26452025-03-17T16:48:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26462025-03-17T16:48:06.602ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26472025-03-17T16:48:06.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26482025-03-17T16:48:06.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26492025-03-17T16:48:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
26502025-03-17T16:48:06.603ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26512025-03-17T16:48:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26522025-03-17T16:48:06.603ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 9.262219ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
26532025-03-17T16:48:06.605ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26542025-03-17T16:48:06.605ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26552025-03-17T16:48:06.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26562025-03-17T16:48:06.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
26572025-03-17T16:48:06.608ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26582025-03-17T16:48:06.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26592025-03-17T16:48:06.611ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26602025-03-17T16:48:06.612ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26612025-03-17T16:48:06.612ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26622025-03-17T16:48:06.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26632025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
26642025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26652025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26662025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
26672025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
26682025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
26692025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
26702025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
26712025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
26722025-03-17T16:48:06.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
26732025-03-17T16:48:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
26742025-03-17T16:48:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
26752025-03-17T16:48:06.620ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26762025-03-17T16:48:06.620ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26772025-03-17T16:48:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26782025-03-17T16:48:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26792025-03-17T16:48:06.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26802025-03-17T16:48:06.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26812025-03-17T16:48:06.623ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26822025-03-17T16:48:06.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26832025-03-17T16:48:06.624ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4173, tv_nsec: 886076699 } })
26842025-03-17T16:48:06.624ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
26852025-03-17T16:48:06.642ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 48.240173ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
26862025-03-17T16:48:06.649ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26872025-03-17T16:48:06.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26882025-03-17T16:48:06.651ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26892025-03-17T16:48:06.652ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26902025-03-17T16:48:06.654ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
26912025-03-17T16:48:06.654ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26922025-03-17T16:48:06.654ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
26932025-03-17T16:48:06.654ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26942025-03-17T16:48:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26952025-03-17T16:48:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26962025-03-17T16:48:06.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26972025-03-17T16:48:06.658ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26982025-03-17T16:48:06.659ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26992025-03-17T16:48:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27002025-03-17T16:48:06.660ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27012025-03-17T16:48:06.660ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27022025-03-17T16:48:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27032025-03-17T16:48:06.661ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27042025-03-17T16:48:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27052025-03-17T16:48:06.662ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.1768ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27062025-03-17T16:48:06.662ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27072025-03-17T16:48:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27082025-03-17T16:48:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
27092025-03-17T16:48:06.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27102025-03-17T16:48:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27112025-03-17T16:48:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
27122025-03-17T16:48:06.664ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27132025-03-17T16:48:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27142025-03-17T16:48:06.664ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27152025-03-17T16:48:06.664ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
27162025-03-17T16:48:06.664ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27172025-03-17T16:48:06.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27182025-03-17T16:48:06.665ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27192025-03-17T16:48:06.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27202025-03-17T16:48:06.666ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27212025-03-17T16:48:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27222025-03-17T16:48:06.670ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27232025-03-17T16:48:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27242025-03-17T16:48:06.670ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
27252025-03-17T16:48:06.670ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
27262025-03-17T16:48:06.670ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 76.356575ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
27272025-03-17T16:48:06.671ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
27282025-03-17T16:48:06.673ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27292025-03-17T16:48:06.676ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27302025-03-17T16:48:06.676ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27312025-03-17T16:48:06.676ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
27322025-03-17T16:48:06.676ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
27332025-03-17T16:48:06.676ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 82.490097ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
27342025-03-17T16:48:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27352025-03-17T16:48:06.678ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 62.519509ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
27362025-03-17T16:48:06.682ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
27372025-03-17T16:48:06.682ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 89.522497ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
27382025-03-17T16:48:06.685ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27392025-03-17T16:48:06.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27402025-03-17T16:48:06.688ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
27412025-03-17T16:48:06.688ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27422025-03-17T16:48:06.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
27432025-03-17T16:48:06.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27442025-03-17T16:48:06.693ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 77.278415ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
27452025-03-17T16:48:06.700ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
27462025-03-17T16:48:06.700ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
27472025-03-17T16:48:06.700ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
27482025-03-17T16:48:06.700ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27492025-03-17T16:48:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27502025-03-17T16:48:06.702ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27512025-03-17T16:48:06.703ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27522025-03-17T16:48:06.704ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27532025-03-17T16:48:06.704ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 110.012506ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27542025-03-17T16:48:06.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27552025-03-17T16:48:06.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27562025-03-17T16:48:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27572025-03-17T16:48:06.705ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27582025-03-17T16:48:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27592025-03-17T16:48:06.706ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27602025-03-17T16:48:06.706ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27612025-03-17T16:48:06.707ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 112.730365ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27622025-03-17T16:48:06.707ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
27632025-03-17T16:48:06.707ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
27642025-03-17T16:48:06.707ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27652025-03-17T16:48:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27662025-03-17T16:48:06.708ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27672025-03-17T16:48:06.709ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
27682025-03-17T16:48:06.709ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27692025-03-17T16:48:06.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27702025-03-17T16:48:06.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27712025-03-17T16:48:06.710ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
27722025-03-17T16:48:06.710ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 115.77848ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
27732025-03-17T16:48:06.710ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27742025-03-17T16:48:06.710ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27752025-03-17T16:48:06.710ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27762025-03-17T16:48:06.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27772025-03-17T16:48:06.711ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
27782025-03-17T16:48:06.711ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
27792025-03-17T16:48:06.711ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
27802025-03-17T16:48:06.711ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27812025-03-17T16:48:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27822025-03-17T16:48:06.712ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27832025-03-17T16:48:06.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27842025-03-17T16:48:06.713ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27852025-03-17T16:48:06.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27862025-03-17T16:48:06.715ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
27872025-03-17T16:48:06.715ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27882025-03-17T16:48:06.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27892025-03-17T16:48:06.719ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
27902025-03-17T16:48:06.719ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
27912025-03-17T16:48:06.719ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
27922025-03-17T16:48:06.719ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 126.027849ms iteration = 8 status = {"generation":1}
27932025-03-17T16:48:06.725ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27942025-03-17T16:48:06.725ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
27952025-03-17T16:48:06.725ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27962025-03-17T16:48:06.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27972025-03-17T16:48:06.737ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27982025-03-17T16:48:06.740ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
27992025-03-17T16:48:06.740ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28002025-03-17T16:48:06.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28012025-03-17T16:48:06.741ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
28022025-03-17T16:48:06.743ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 149.42579ms iteration = 15 status = 0
28032025-03-17T16:48:06.750ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
28042025-03-17T16:48:06.750ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28052025-03-17T16:48:06.750ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4753b48c-7032-42b6-9f0e-9953289bf841", "content-length": "26", "date": "Mon, 17 Mar 2025 16:48:06 GMT"} })
28062025-03-17T16:48:06.750ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28072025-03-17T16:48:06.751ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
28082025-03-17T16:48:06.751ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 156.8906ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
28092025-03-17T16:48:06.752ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28102025-03-17T16:48:06.754ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 159.6668ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
28112025-03-17T16:48:06.756ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28122025-03-17T16:48:06.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28132025-03-17T16:48:06.759ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
28142025-03-17T16:48:06.759ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
28152025-03-17T16:48:06.759ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28162025-03-17T16:48:06.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28172025-03-17T16:48:06.767ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28182025-03-17T16:48:06.767ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
28192025-03-17T16:48:06.767ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 172.835182ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
28202025-03-17T16:48:06.769ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
28212025-03-17T16:48:06.769ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
28222025-03-17T16:48:06.769ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 176.312486ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
28232025-03-17T16:48:06.769ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 175.475169ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
28242025-03-17T16:48:06.785ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
28252025-03-17T16:48:06.785ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28262025-03-17T16:48:06.785ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d662830-a093-4fc3-a4bd-9b5a8cdcafe3", "content-length": "26", "date": "Mon, 17 Mar 2025 16:48:06 GMT"} })
28272025-03-17T16:48:06.785ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28282025-03-17T16:48:06.787ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
28292025-03-17T16:48:06.794ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 200.334881ms iteration = 16 status = {"physical_disks_added":0}
28302025-03-17T16:48:06.857ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
28312025-03-17T16:48:06.889ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
28322025-03-17T16:48:06.889ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28332025-03-17T16:48:06.889ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9414fd2a-f2ad-47e2-a855-eafe7a25a0f3", "content-length": "26", "date": "Mon, 17 Mar 2025 16:48:06 GMT"} })
28342025-03-17T16:48:06.889ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28352025-03-17T16:48:06.892ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28362025-03-17T16:48:06.929ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
28372025-03-17T16:48:06.933ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 339.829028ms iteration = 15 status = {}
28382025-03-17T16:48:07.033ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
28392025-03-17T16:48:07.034ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
28402025-03-17T16:48:07.034ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a71cf9da-af79-4e07-97dc-faded0a28726", "content-length": "372", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} })
28412025-03-17T16:48:07.034ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28422025-03-17T16:48:07.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28432025-03-17T16:48:07.036ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28442025-03-17T16:48:07.036ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28452025-03-17T16:48:07.038ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28462025-03-17T16:48:07.038ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28472025-03-17T16:48:07.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28482025-03-17T16:48:07.039ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
28492025-03-17T16:48:07.039ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28502025-03-17T16:48:07.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28512025-03-17T16:48:07.041ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
28522025-03-17T16:48:07.042ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
28532025-03-17T16:48:07.042ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
28542025-03-17T16:48:07.042ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 448.811542ms iteration = 8 status = {"generation":2}
28552025-03-17T16:48:07.069ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28562025-03-17T16:48:07.071ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28572025-03-17T16:48:07.071ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28582025-03-17T16:48:07.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28592025-03-17T16:48:07.079ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
28602025-03-17T16:48:07.079ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
28612025-03-17T16:48:07.079ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "5a980446-fed1-4a22-9154-726d7b409407", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} })
28622025-03-17T16:48:07.079ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28632025-03-17T16:48:07.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28642025-03-17T16:48:07.081ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28652025-03-17T16:48:07.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28662025-03-17T16:48:07.084ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28672025-03-17T16:48:07.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28682025-03-17T16:48:07.089ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
28692025-03-17T16:48:07.089ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "355315a1-bf51-4587-a312-ea0b5a20b77a", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
28702025-03-17T16:48:07.089ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
28712025-03-17T16:48:07.091ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "b6fd76fa-2a0b-4c3a-8962-197fd3467f54", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
28722025-03-17T16:48:07.091ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
28732025-03-17T16:48:07.092ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "46d751c2-7922-47ff-a7e3-415b59adc053", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
28742025-03-17T16:48:07.092ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
28752025-03-17T16:48:07.094ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
28762025-03-17T16:48:07.094ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 500.926428ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
28772025-03-17T16:48:07.146ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
28782025-03-17T16:48:07.146ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "f30a21db-064e-4022-a5b9-a529914086ed", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
28792025-03-17T16:48:07.146ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
28802025-03-17T16:48:07.148ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "e3b4f778-8d3c-40fd-91e8-7d99075a6fc0", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
28812025-03-17T16:48:07.148ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
28822025-03-17T16:48:07.150ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
28832025-03-17T16:48:07.150ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 557.330497ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
28842025-03-17T16:48:07.157ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
28852025-03-17T16:48:07.157ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b5f075-7687-4e1a-8a9d-0effd221bb9e", "content-length": "1", "date": "Mon, 17 Mar 2025 16:48:06 GMT"} })
28862025-03-17T16:48:07.160ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
28872025-03-17T16:48:07.160ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
28882025-03-17T16:48:07.176ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
28892025-03-17T16:48:07.177ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "917c76d7-1487-432e-b153-11da1f3c9cb8", "content-length": "691", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} })
28902025-03-17T16:48:07.177ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
28912025-03-17T16:48:07.177ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
28922025-03-17T16:48:07.177ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
28932025-03-17T16:48:07.178ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 584.345395ms iteration = 15 status = {}
28942025-03-17T16:48:07.179ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 586.120873ms iteration = 15 status = null
28952025-03-17T16:48:07.179ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
28962025-03-17T16:48:07.179ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
28972025-03-17T16:48:07.180ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "7af63dfd-81ca-4b85-bf62-e001033d1d32", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} })
28982025-03-17T16:48:07.181ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 587.86217ms iteration = 15 status = {}
28992025-03-17T16:48:07.200ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29002025-03-17T16:48:07.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29012025-03-17T16:48:07.202ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
29022025-03-17T16:48:07.202ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29032025-03-17T16:48:07.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29042025-03-17T16:48:07.212ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
29052025-03-17T16:48:07.212ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "47a9d11e-0944-401f-aab1-25d806b8ae39", "content-length": "134", "date": "Mon, 17 Mar 2025 16:48:06 GMT"} })
29062025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
29072025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
29082025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
29092025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
29102025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29112025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29122025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29132025-03-17T16:48:07.212ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
29142025-03-17T16:48:07.213ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29152025-03-17T16:48:07.213ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
29162025-03-17T16:48:07.213ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
29172025-03-17T16:48:07.213ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29182025-03-17T16:48:07.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29192025-03-17T16:48:07.215ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
29202025-03-17T16:48:07.215ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "945a5c64-4979-41d7-a251-5dbf8a1da722", "content-length": "2", "date": "Mon, 17 Mar 2025 16:48:06 GMT"} })
29212025-03-17T16:48:07.215ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
29222025-03-17T16:48:07.215ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f3665bb-f9a2-4a72-8d9a-6a0af88ce5b8", "content-length": "52", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} })
29232025-03-17T16:48:07.215ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
29242025-03-17T16:48:07.215ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
29252025-03-17T16:48:07.307ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "43ed3e81-01f9-41e7-a655-99688c0dd10e", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} })
29262025-03-17T16:48:07.307ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
29272025-03-17T16:48:07.307ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
29282025-03-17T16:48:07.307ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f9c9f27-4ce9-48f7-a3cc-7b2f76101eee", "content-length": "682", "date": "Mon, 17 Mar 2025 16:48:07 GMT"} })
29292025-03-17T16:48:07.307ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29302025-03-17T16:48:07.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29312025-03-17T16:48:07.312ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29322025-03-17T16:48:07.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29332025-03-17T16:48:07.316ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29342025-03-17T16:48:07.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29352025-03-17T16:48:07.319ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29362025-03-17T16:48:07.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29372025-03-17T16:48:07.326ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 732.088019ms iteration = 15 status = {}
29382025-03-17T16:48:10.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 061a72ae-2738-4214-83d0-886f3b655d20 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
29392025-03-17T16:48:16.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
29402025-03-17T16:48:16.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29412025-03-17T16:48:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29422025-03-17T16:48:16.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
29432025-03-17T16:48:16.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29442025-03-17T16:48:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29452025-03-17T16:48:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29462025-03-17T16:48:16.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29472025-03-17T16:48:16.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29482025-03-17T16:48:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29492025-03-17T16:48:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29502025-03-17T16:48:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29512025-03-17T16:48:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29522025-03-17T16:48:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29532025-03-17T16:48:16.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29542025-03-17T16:48:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
29552025-03-17T16:48:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29562025-03-17T16:48:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29572025-03-17T16:48:16.613ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
29582025-03-17T16:48:16.613ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.779415ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
29592025-03-17T16:48:16.632ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.249759ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29602025-03-17T16:48:20.125ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:39341
29612025-03-17T16:48:20.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 4701b0e4-bce8-4121-a8d7-1af891e601f6 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
29622025-03-17T16:48:20.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 4701b0e4-bce8-4121-a8d7-1af891e601f6 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
29632025-03-17T16:48:20.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2196 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 4701b0e4-bce8-4121-a8d7-1af891e601f6 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
29642025-03-17T16:48:20.127ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = bf4dfa44-8c3d-46bc-962f-67f42451b620 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
29652025-03-17T16:48:20.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = bf4dfa44-8c3d-46bc-962f-67f42451b620 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
29662025-03-17T16:48:20.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2325 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = bf4dfa44-8c3d-46bc-962f-67f42451b620 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
29672025-03-17T16:48:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = f60e14b2-15d4-4981-865d-6f8b1cf934ec response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
29682025-03-17T16:48:26.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
29692025-03-17T16:48:26.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29702025-03-17T16:48:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29712025-03-17T16:48:26.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
29722025-03-17T16:48:26.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29732025-03-17T16:48:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29742025-03-17T16:48:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29752025-03-17T16:48:26.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29762025-03-17T16:48:26.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29772025-03-17T16:48:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
29782025-03-17T16:48:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29792025-03-17T16:48:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29802025-03-17T16:48:26.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29812025-03-17T16:48:26.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29822025-03-17T16:48:26.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29832025-03-17T16:48:26.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
29842025-03-17T16:48:26.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29852025-03-17T16:48:26.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29862025-03-17T16:48:26.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
29872025-03-17T16:48:26.614ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.736413ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
29882025-03-17T16:48:26.634ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.550149ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29892025-03-17T16:48:30.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 1c4d16c7-b62c-496e-b633-4bb63676f946 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
29902025-03-17T16:48:35.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 39344e81-5744-4503-81aa-0d8341237d32 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
29912025-03-17T16:48:35.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 39344e81-5744-4503-81aa-0d8341237d32 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
29922025-03-17T16:48:35.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2561 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 39344e81-5744-4503-81aa-0d8341237d32 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
29932025-03-17T16:48:35.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 77dc2de7-4982-426f-bd00-bf5c2d438cd2 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
29942025-03-17T16:48:35.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 77dc2de7-4982-426f-bd00-bf5c2d438cd2 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
29952025-03-17T16:48:35.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2102 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 77dc2de7-4982-426f-bd00-bf5c2d438cd2 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
29962025-03-17T16:48:36.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
29972025-03-17T16:48:36.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
29982025-03-17T16:48:36.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
29992025-03-17T16:48:36.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
30002025-03-17T16:48:36.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30012025-03-17T16:48:36.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
30022025-03-17T16:48:36.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
30032025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30042025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
30052025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
30062025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
30072025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
30082025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30092025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30102025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
30112025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
30122025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
30132025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30142025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
30152025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
30162025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
30172025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
30182025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
30192025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30202025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30212025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
30222025-03-17T16:48:36.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
30232025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
30242025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30252025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30262025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
30272025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30282025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
30292025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
30302025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
30312025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
30322025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
30332025-03-17T16:48:36.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30342025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30352025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30362025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30372025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
30382025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
30392025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30402025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
30412025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30422025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
30432025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30442025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30452025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30462025-03-17T16:48:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30472025-03-17T16:48:36.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30482025-03-17T16:48:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4203, tv_nsec: 859393737 } })
30492025-03-17T16:48:36.597ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30502025-03-17T16:48:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
30512025-03-17T16:48:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4203, tv_nsec: 859783195 } })
30522025-03-17T16:48:36.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.070157ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
30532025-03-17T16:48:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
30542025-03-17T16:48:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30552025-03-17T16:48:36.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30562025-03-17T16:48:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30572025-03-17T16:48:36.599ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30582025-03-17T16:48:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30592025-03-17T16:48:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30602025-03-17T16:48:36.602ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30612025-03-17T16:48:36.603ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30622025-03-17T16:48:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
30632025-03-17T16:48:36.604ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
30642025-03-17T16:48:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
30652025-03-17T16:48:36.606ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30662025-03-17T16:48:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30672025-03-17T16:48:36.607ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30682025-03-17T16:48:36.607ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30692025-03-17T16:48:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
30702025-03-17T16:48:36.607ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30712025-03-17T16:48:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30722025-03-17T16:48:36.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30732025-03-17T16:48:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30742025-03-17T16:48:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
30752025-03-17T16:48:36.610ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30762025-03-17T16:48:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30772025-03-17T16:48:36.610ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
30782025-03-17T16:48:36.610ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30792025-03-17T16:48:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30802025-03-17T16:48:36.615ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30812025-03-17T16:48:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30822025-03-17T16:48:36.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30832025-03-17T16:48:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30842025-03-17T16:48:36.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
30852025-03-17T16:48:36.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
30862025-03-17T16:48:36.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
30872025-03-17T16:48:36.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
30882025-03-17T16:48:36.616ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30892025-03-17T16:48:36.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
30902025-03-17T16:48:36.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
30912025-03-17T16:48:36.617ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30922025-03-17T16:48:36.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30932025-03-17T16:48:36.618ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30942025-03-17T16:48:36.620ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
30952025-03-17T16:48:36.620ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30962025-03-17T16:48:36.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30972025-03-17T16:48:36.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
30982025-03-17T16:48:36.621ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30992025-03-17T16:48:36.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31002025-03-17T16:48:36.643ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31012025-03-17T16:48:36.643ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
31022025-03-17T16:48:36.643ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31032025-03-17T16:48:36.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
31042025-03-17T16:48:36.648ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31052025-03-17T16:48:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31062025-03-17T16:48:36.649ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
31072025-03-17T16:48:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
31082025-03-17T16:48:36.649ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31092025-03-17T16:48:36.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31102025-03-17T16:48:36.650ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 56.122062ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31112025-03-17T16:48:36.653ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31122025-03-17T16:48:36.653ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
31132025-03-17T16:48:36.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
31142025-03-17T16:48:36.656ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
31152025-03-17T16:48:36.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31162025-03-17T16:48:36.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31172025-03-17T16:48:36.657ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31182025-03-17T16:48:36.657ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31192025-03-17T16:48:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31202025-03-17T16:48:36.659ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 64.234596ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
31212025-03-17T16:48:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
31222025-03-17T16:48:36.660ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31232025-03-17T16:48:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31242025-03-17T16:48:36.664ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31252025-03-17T16:48:36.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31262025-03-17T16:48:36.666ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31272025-03-17T16:48:36.667ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4203, tv_nsec: 929087029 } })
31282025-03-17T16:48:36.667ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 50.494121ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
31292025-03-17T16:48:36.667ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
31302025-03-17T16:48:36.667ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
31312025-03-17T16:48:36.667ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 73.930524ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
31322025-03-17T16:48:36.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
31332025-03-17T16:48:36.669ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31342025-03-17T16:48:36.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31352025-03-17T16:48:36.670ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
31362025-03-17T16:48:36.670ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 75.625179ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
31372025-03-17T16:48:36.671ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 76.499649ms iteration = 16 status = 0
31382025-03-17T16:48:36.671ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 76.535719ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
31392025-03-17T16:48:36.673ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 56.447799ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
31402025-03-17T16:48:36.684ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31412025-03-17T16:48:36.684ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
31422025-03-17T16:48:36.684ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31432025-03-17T16:48:36.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
31442025-03-17T16:48:36.690ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31452025-03-17T16:48:36.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
31462025-03-17T16:48:36.693ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31472025-03-17T16:48:36.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31482025-03-17T16:48:36.700ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 105.290955ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
31492025-03-17T16:48:36.700ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 105.587671ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
31502025-03-17T16:48:36.717ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31512025-03-17T16:48:36.717ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
31522025-03-17T16:48:36.717ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 122.8098ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
31532025-03-17T16:48:36.723ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
31542025-03-17T16:48:36.723ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 128.156815ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
31552025-03-17T16:48:36.727ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 132.525778ms iteration = 17 status = {"physical_disks_added":0}
31562025-03-17T16:48:36.729ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31572025-03-17T16:48:36.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31582025-03-17T16:48:36.744ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
31592025-03-17T16:48:36.744ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31602025-03-17T16:48:36.744ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e37bac5-5fd1-4fa0-9b9c-855d0d01ae77", "content-length": "26", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
31612025-03-17T16:48:36.744ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31622025-03-17T16:48:36.746ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31632025-03-17T16:48:36.769ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
31642025-03-17T16:48:36.769ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31652025-03-17T16:48:36.770ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "431e5980-4f8d-40c6-b794-d940758ffaac", "content-length": "26", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
31662025-03-17T16:48:36.770ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31672025-03-17T16:48:36.772ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
31682025-03-17T16:48:36.783ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
31692025-03-17T16:48:36.783ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31702025-03-17T16:48:36.784ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae656872-1174-463b-9405-880de57d6211", "content-length": "26", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
31712025-03-17T16:48:36.784ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31722025-03-17T16:48:36.785ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31732025-03-17T16:48:36.823ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31742025-03-17T16:48:36.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31752025-03-17T16:48:36.825ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31762025-03-17T16:48:36.828ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
31772025-03-17T16:48:36.828ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31782025-03-17T16:48:36.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31792025-03-17T16:48:36.829ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31802025-03-17T16:48:36.831ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
31812025-03-17T16:48:36.831ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31822025-03-17T16:48:36.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31832025-03-17T16:48:36.928ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
31842025-03-17T16:48:36.928ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31852025-03-17T16:48:36.929ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b54679c4-7a51-431f-950c-f4a1fc72389d", "content-length": "372", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
31862025-03-17T16:48:36.929ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31872025-03-17T16:48:36.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31882025-03-17T16:48:36.933ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
31892025-03-17T16:48:36.933ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbaee942-2e56-41c6-b5fb-b73c0a943996", "content-length": "1", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
31902025-03-17T16:48:36.934ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
31912025-03-17T16:48:36.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
31922025-03-17T16:48:36.936ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 342.685759ms iteration = 16 status = null
31932025-03-17T16:48:36.966ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 373.235273ms iteration = 16 status = {}
31942025-03-17T16:48:36.977ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
31952025-03-17T16:48:36.977ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
31962025-03-17T16:48:36.977ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "00708f18-7db9-4f77-aaa7-1bf35b52bb4c", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
31972025-03-17T16:48:36.977ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31982025-03-17T16:48:36.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31992025-03-17T16:48:37.037ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
32002025-03-17T16:48:37.038ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "86e8c10e-639e-4b83-8338-fabedc51e42b", "content-length": "691", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
32012025-03-17T16:48:37.038ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
32022025-03-17T16:48:37.038ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
32032025-03-17T16:48:37.038ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
32042025-03-17T16:48:37.039ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 444.803226ms iteration = 16 status = {}
32052025-03-17T16:48:37.042ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32062025-03-17T16:48:37.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32072025-03-17T16:48:37.045ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32082025-03-17T16:48:37.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32092025-03-17T16:48:37.048ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
32102025-03-17T16:48:37.048ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
32112025-03-17T16:48:37.048ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "35f2d7b6-cda7-43c9-aec7-cf5d20e8ced7", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
32122025-03-17T16:48:37.050ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 456.093568ms iteration = 16 status = {}
32132025-03-17T16:48:37.080ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32142025-03-17T16:48:37.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32152025-03-17T16:48:37.082ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
32162025-03-17T16:48:37.082ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32172025-03-17T16:48:37.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32182025-03-17T16:48:37.093ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
32192025-03-17T16:48:37.094ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f612b5c-59bb-483d-a161-960aa940c4dc", "content-length": "134", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
32202025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
32212025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
32222025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
32232025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
32242025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32252025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32262025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32272025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32282025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32292025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32302025-03-17T16:48:37.094ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
32312025-03-17T16:48:37.094ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32322025-03-17T16:48:37.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32332025-03-17T16:48:37.096ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
32342025-03-17T16:48:37.097ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "5639bb8e-9449-4a0b-a297-d34d3b1a2e4c", "content-length": "2", "date": "Mon, 17 Mar 2025 16:48:37 GMT"} })
32352025-03-17T16:48:37.097ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
32362025-03-17T16:48:37.097ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fd57bd7-66b0-4f83-94df-6c1a1a2fdc39", "content-length": "52", "date": "Mon, 17 Mar 2025 16:48:37 GMT"} })
32372025-03-17T16:48:37.097ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
32382025-03-17T16:48:37.097ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
32392025-03-17T16:48:37.188ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "c8a5f9d6-bf36-4e5b-b0bf-de289e5e23a3", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
32402025-03-17T16:48:37.188ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
32412025-03-17T16:48:37.188ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
32422025-03-17T16:48:37.190ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "23c3209a-b3cc-4cbc-9dc2-a9701cec1f5a", "content-length": "682", "date": "Mon, 17 Mar 2025 16:48:36 GMT"} })
32432025-03-17T16:48:37.190ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32442025-03-17T16:48:37.191ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32452025-03-17T16:48:37.195ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32462025-03-17T16:48:37.195ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32472025-03-17T16:48:37.198ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32482025-03-17T16:48:37.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32492025-03-17T16:48:37.201ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32502025-03-17T16:48:37.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32512025-03-17T16:48:37.208ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 614.041181ms iteration = 16 status = {}
32522025-03-17T16:48:40.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 421 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 65f1843d-1c9b-47a4-b086-6b96116afd13 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
32532025-03-17T16:48:46.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
32542025-03-17T16:48:46.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32552025-03-17T16:48:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32562025-03-17T16:48:46.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
32572025-03-17T16:48:46.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32582025-03-17T16:48:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32592025-03-17T16:48:46.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32602025-03-17T16:48:46.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32612025-03-17T16:48:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32622025-03-17T16:48:46.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32632025-03-17T16:48:46.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32642025-03-17T16:48:46.599ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.620109ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
32652025-03-17T16:48:46.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32662025-03-17T16:48:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32672025-03-17T16:48:46.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32682025-03-17T16:48:46.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32692025-03-17T16:48:46.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
32702025-03-17T16:48:46.607ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32712025-03-17T16:48:46.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32722025-03-17T16:48:46.617ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
32732025-03-17T16:48:46.617ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.905152ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
32742025-03-17T16:48:50.124ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 052c6ee6-b602-4b02-8eac-ad67c1c11fed roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
32752025-03-17T16:48:50.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 052c6ee6-b602-4b02-8eac-ad67c1c11fed resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
32762025-03-17T16:48:50.128ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3594 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 052c6ee6-b602-4b02-8eac-ad67c1c11fed response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
32772025-03-17T16:48:50.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 288f744b-e648-4bfb-b494-06fa7b30e508 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
32782025-03-17T16:48:50.129ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 288f744b-e648-4bfb-b494-06fa7b30e508 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
32792025-03-17T16:48:50.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2076 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 288f744b-e648-4bfb-b494-06fa7b30e508 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
32802025-03-17T16:48:50.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 284 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 41dcafd9-ffb1-4a15-8ee6-7d5a7043be6b response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
32812025-03-17T16:48:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
32822025-03-17T16:48:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32832025-03-17T16:48:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32842025-03-17T16:48:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
32852025-03-17T16:48:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32862025-03-17T16:48:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32872025-03-17T16:48:56.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32882025-03-17T16:48:56.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32892025-03-17T16:48:56.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32902025-03-17T16:48:56.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32912025-03-17T16:48:56.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32922025-03-17T16:48:56.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32932025-03-17T16:48:56.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32942025-03-17T16:48:56.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
32952025-03-17T16:48:56.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32962025-03-17T16:48:56.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32972025-03-17T16:48:56.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
32982025-03-17T16:48:56.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.701104ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
32992025-03-17T16:48:56.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33002025-03-17T16:48:56.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33012025-03-17T16:48:56.624ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.030624ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33022025-03-17T16:49:00.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 252 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 89905c92-f838-4725-b84e-e18c8c4de464 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
33032025-03-17T16:49:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
33042025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
33052025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
33062025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
33072025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
33082025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33092025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
33102025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
33112025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
33122025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
33132025-03-17T16:49:06.593ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:39:06.593064261 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
33142025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
33152025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
33162025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
33172025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
33182025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
33192025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
33202025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
33212025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
33222025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
33232025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
33242025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
33252025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
33262025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
33272025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33282025-03-17T16:49:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4233, tv_nsec: 855815728 } })
33292025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
33302025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
33312025-03-17T16:49:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4233, tv_nsec: 855838339 } })
33322025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
33332025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
33342025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
33352025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
33362025-03-17T16:49:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
33372025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
33382025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
33392025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
33402025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
33412025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 15.38\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
33422025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
33432025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
33442025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
33452025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
33462025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
33472025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
33482025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33492025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33502025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
33512025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33522025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
33532025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
33542025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
33552025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
33562025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
33572025-03-17T16:49:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
33582025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
33592025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33602025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33612025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33622025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
33632025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
33642025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
33652025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33662025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
33672025-03-17T16:49:06.594ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
33682025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
33692025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
33702025-03-17T16:49:06.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
33712025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33722025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
33732025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33742025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
33752025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33762025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
33772025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33782025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
33792025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33802025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33812025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33822025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
33832025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.337709ms iteration = 9 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
33842025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
33852025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33862025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33872025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
33882025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33892025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.378639ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
33902025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
33912025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
33922025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33932025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33942025-03-17T16:49:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33952025-03-17T16:49:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4233, tv_nsec: 858219650 } })
33962025-03-17T16:49:06.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
33972025-03-17T16:49:06.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
33982025-03-17T16:49:06.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33992025-03-17T16:49:06.596ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:39:06.593064261 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
34002025-03-17T16:49:06.596ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.464694ms iteration = 9 status = {"expiration":"2025-03-17T16:39:06.593064261Z","pruned":[]}
34012025-03-17T16:49:06.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
34022025-03-17T16:49:06.596ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.75014ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
34032025-03-17T16:49:06.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
34042025-03-17T16:49:06.597ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34052025-03-17T16:49:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
34062025-03-17T16:49:06.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4233, tv_nsec: 860094170 } })
34072025-03-17T16:49:06.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
34082025-03-17T16:49:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34092025-03-17T16:49:06.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34102025-03-17T16:49:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34112025-03-17T16:49:06.601ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
34122025-03-17T16:49:06.601ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34132025-03-17T16:49:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
34142025-03-17T16:49:06.604ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34152025-03-17T16:49:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
34162025-03-17T16:49:06.607ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34172025-03-17T16:49:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34182025-03-17T16:49:06.611ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34192025-03-17T16:49:06.611ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 17.954104ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
34202025-03-17T16:49:06.614ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
34212025-03-17T16:49:06.614ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34222025-03-17T16:49:06.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34232025-03-17T16:49:06.615ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34242025-03-17T16:49:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34252025-03-17T16:49:06.616ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34262025-03-17T16:49:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34272025-03-17T16:49:06.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
34282025-03-17T16:49:06.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
34292025-03-17T16:49:06.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
34302025-03-17T16:49:06.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
34312025-03-17T16:49:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
34322025-03-17T16:49:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
34332025-03-17T16:49:06.617ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34342025-03-17T16:49:06.618ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34352025-03-17T16:49:06.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34362025-03-17T16:49:06.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
34372025-03-17T16:49:06.638ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 44.170695ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
34382025-03-17T16:49:06.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34392025-03-17T16:49:06.642ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34402025-03-17T16:49:06.642ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
34412025-03-17T16:49:06.642ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34422025-03-17T16:49:06.642ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34432025-03-17T16:49:06.642ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
34442025-03-17T16:49:06.642ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34452025-03-17T16:49:06.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34462025-03-17T16:49:06.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34472025-03-17T16:49:06.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34482025-03-17T16:49:06.644ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34492025-03-17T16:49:06.644ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34502025-03-17T16:49:06.648ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34512025-03-17T16:49:06.648ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
34522025-03-17T16:49:06.648ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34532025-03-17T16:49:06.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34542025-03-17T16:49:06.650ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
34552025-03-17T16:49:06.650ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34562025-03-17T16:49:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
34572025-03-17T16:49:06.650ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
34582025-03-17T16:49:06.650ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
34592025-03-17T16:49:06.650ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 56.9376ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
34602025-03-17T16:49:06.650ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34612025-03-17T16:49:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34622025-03-17T16:49:06.651ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34632025-03-17T16:49:06.651ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
34642025-03-17T16:49:06.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34652025-03-17T16:49:06.652ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
34662025-03-17T16:49:06.652ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
34672025-03-17T16:49:06.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
34682025-03-17T16:49:06.653ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34692025-03-17T16:49:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
34702025-03-17T16:49:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
34712025-03-17T16:49:06.656ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34722025-03-17T16:49:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34732025-03-17T16:49:06.657ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34742025-03-17T16:49:06.658ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34752025-03-17T16:49:06.659ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
34762025-03-17T16:49:06.659ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34772025-03-17T16:49:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34782025-03-17T16:49:06.660ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4233, tv_nsec: 922492676 } })
34792025-03-17T16:49:06.660ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
34802025-03-17T16:49:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34812025-03-17T16:49:06.660ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34822025-03-17T16:49:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34832025-03-17T16:49:06.662ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34842025-03-17T16:49:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
34852025-03-17T16:49:06.662ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34862025-03-17T16:49:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
34872025-03-17T16:49:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
34882025-03-17T16:49:06.663ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
34892025-03-17T16:49:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
34902025-03-17T16:49:06.665ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
34912025-03-17T16:49:06.665ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34922025-03-17T16:49:06.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34932025-03-17T16:49:06.665ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34942025-03-17T16:49:06.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34952025-03-17T16:49:06.666ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34962025-03-17T16:49:06.666ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
34972025-03-17T16:49:06.666ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34982025-03-17T16:49:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34992025-03-17T16:49:06.668ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 74.719751ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
35002025-03-17T16:49:06.672ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35012025-03-17T16:49:06.672ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
35022025-03-17T16:49:06.672ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 79.127494ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
35032025-03-17T16:49:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35042025-03-17T16:49:06.674ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 80.952494ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
35052025-03-17T16:49:06.676ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35062025-03-17T16:49:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
35072025-03-17T16:49:06.679ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35082025-03-17T16:49:06.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35092025-03-17T16:49:06.694ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
35102025-03-17T16:49:06.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
35112025-03-17T16:49:06.696ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35122025-03-17T16:49:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35132025-03-17T16:49:06.699ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 82.817084ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
35142025-03-17T16:49:06.700ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35152025-03-17T16:49:06.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35162025-03-17T16:49:06.702ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35172025-03-17T16:49:06.703ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 109.573737ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
35182025-03-17T16:49:06.705ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
35192025-03-17T16:49:06.705ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
35202025-03-17T16:49:06.705ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
35212025-03-17T16:49:06.705ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35222025-03-17T16:49:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35232025-03-17T16:49:06.706ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
35242025-03-17T16:49:06.706ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35252025-03-17T16:49:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35262025-03-17T16:49:06.707ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
35272025-03-17T16:49:06.707ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
35282025-03-17T16:49:06.707ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
35292025-03-17T16:49:06.707ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35302025-03-17T16:49:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35312025-03-17T16:49:06.707ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35322025-03-17T16:49:06.708ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 91.51049ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
35332025-03-17T16:49:06.709ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 115.464193ms iteration = 17 status = 0
35342025-03-17T16:49:06.711ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
35352025-03-17T16:49:06.711ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
35362025-03-17T16:49:06.711ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 117.588088ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
35372025-03-17T16:49:06.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
35382025-03-17T16:49:06.712ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35392025-03-17T16:49:06.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35402025-03-17T16:49:06.713ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35412025-03-17T16:49:06.713ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35422025-03-17T16:49:06.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35432025-03-17T16:49:06.714ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
35442025-03-17T16:49:06.714ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
35452025-03-17T16:49:06.714ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35462025-03-17T16:49:06.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35472025-03-17T16:49:06.715ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
35482025-03-17T16:49:06.715ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 121.182396ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
35492025-03-17T16:49:06.715ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 121.149955ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
35502025-03-17T16:49:06.716ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35512025-03-17T16:49:06.716ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35522025-03-17T16:49:06.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35532025-03-17T16:49:06.717ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35542025-03-17T16:49:06.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35552025-03-17T16:49:06.718ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35562025-03-17T16:49:06.718ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
35572025-03-17T16:49:06.718ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
35582025-03-17T16:49:06.718ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 124.473306ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
35592025-03-17T16:49:06.718ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35602025-03-17T16:49:06.718ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35612025-03-17T16:49:06.719ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35622025-03-17T16:49:06.721ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
35632025-03-17T16:49:06.721ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35642025-03-17T16:49:06.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35652025-03-17T16:49:06.722ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
35662025-03-17T16:49:06.722ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35672025-03-17T16:49:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35682025-03-17T16:49:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
35692025-03-17T16:49:06.722ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35702025-03-17T16:49:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35712025-03-17T16:49:06.723ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
35722025-03-17T16:49:06.723ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35732025-03-17T16:49:06.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35742025-03-17T16:49:06.724ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
35752025-03-17T16:49:06.724ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 130.917404ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
35762025-03-17T16:49:06.726ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 132.105059ms iteration = 18 status = {"physical_disks_added":0}
35772025-03-17T16:49:06.726ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
35782025-03-17T16:49:06.726ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
35792025-03-17T16:49:06.726ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
35802025-03-17T16:49:06.726ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 133.710784ms iteration = 9 status = {"generation":2}
35812025-03-17T16:49:06.748ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35822025-03-17T16:49:06.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
35832025-03-17T16:49:06.751ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35842025-03-17T16:49:06.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35852025-03-17T16:49:06.766ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
35862025-03-17T16:49:06.766ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 172.26673ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
35872025-03-17T16:49:06.771ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
35882025-03-17T16:49:06.771ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
35892025-03-17T16:49:06.772ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d48def70-a985-490d-82ba-d19807a32cc6", "content-length": "26", "date": "Mon, 17 Mar 2025 16:49:06 GMT"} })
35902025-03-17T16:49:06.772ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
35912025-03-17T16:49:06.774ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
35922025-03-17T16:49:06.777ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
35932025-03-17T16:49:06.777ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 184.756747ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
35942025-03-17T16:49:06.781ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
35952025-03-17T16:49:06.781ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
35962025-03-17T16:49:06.781ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35972025-03-17T16:49:06.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35982025-03-17T16:49:06.783ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35992025-03-17T16:49:06.786ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
36002025-03-17T16:49:06.786ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36012025-03-17T16:49:06.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36022025-03-17T16:49:06.789ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
36032025-03-17T16:49:06.789ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
36042025-03-17T16:49:06.789ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
36052025-03-17T16:49:06.789ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 196.283963ms iteration = 9 status = {"generation":1}
36062025-03-17T16:49:06.927ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
36072025-03-17T16:49:06.927ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
36082025-03-17T16:49:06.927ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8764956-b221-4116-91f8-9a942020da65", "content-length": "26", "date": "Mon, 17 Mar 2025 16:49:06 GMT"} })
36092025-03-17T16:49:06.927ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
36102025-03-17T16:49:06.929ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
36112025-03-17T16:49:06.988ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 395.398089ms iteration = 17 status = {}
36122025-03-17T16:49:06.991ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
36132025-03-17T16:49:07.004ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
36142025-03-17T16:49:07.004ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a512ef8-7439-45b6-bfee-f0db2137156f", "content-length": "691", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} })
36152025-03-17T16:49:07.004ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
36162025-03-17T16:49:07.004ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
36172025-03-17T16:49:07.004ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
36182025-03-17T16:49:07.007ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 413.179689ms iteration = 17 status = {}
36192025-03-17T16:49:07.012ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
36202025-03-17T16:49:07.012ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
36212025-03-17T16:49:07.013ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c345439-f2bf-4457-8bcc-2b2429e286d3", "content-length": "26", "date": "Mon, 17 Mar 2025 16:49:06 GMT"} })
36222025-03-17T16:49:07.013ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
36232025-03-17T16:49:07.014ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
36242025-03-17T16:49:07.020ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
36252025-03-17T16:49:07.025ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
36262025-03-17T16:49:07.025ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
36272025-03-17T16:49:07.025ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8264d649-e9a2-4b9b-a84e-1aea459852b5", "content-length": "372", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} })
36282025-03-17T16:49:07.025ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36292025-03-17T16:49:07.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36302025-03-17T16:49:07.040ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
36312025-03-17T16:49:07.040ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
36322025-03-17T16:49:07.040ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "da509357-132c-45cd-8bda-85958a159211", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} })
36332025-03-17T16:49:07.040ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36342025-03-17T16:49:07.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36352025-03-17T16:49:07.043ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36362025-03-17T16:49:07.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36372025-03-17T16:49:07.046ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36382025-03-17T16:49:07.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36392025-03-17T16:49:07.050ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
36402025-03-17T16:49:07.050ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
36412025-03-17T16:49:07.050ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "8fd37c99-2bd7-49d0-ba2e-d577fdd76971", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} })
36422025-03-17T16:49:07.052ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 458.827116ms iteration = 17 status = {}
36432025-03-17T16:49:07.129ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
36442025-03-17T16:49:07.129ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ccad9f8-4fe1-4153-a110-afef1257ffcb", "content-length": "1", "date": "Mon, 17 Mar 2025 16:49:06 GMT"} })
36452025-03-17T16:49:07.130ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
36462025-03-17T16:49:07.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
36472025-03-17T16:49:07.133ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 539.918774ms iteration = 17 status = null
36482025-03-17T16:49:07.225ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
36492025-03-17T16:49:07.238ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
36502025-03-17T16:49:07.239ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "01140d9a-282e-473c-937c-2e06cf1e805d", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
36512025-03-17T16:49:07.239ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
36522025-03-17T16:49:07.240ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "8b5c0d5e-f30d-4b62-959f-9e0d6d79eef1", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
36532025-03-17T16:49:07.240ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
36542025-03-17T16:49:07.242ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
36552025-03-17T16:49:07.242ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 649.47694ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
36562025-03-17T16:49:07.254ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36572025-03-17T16:49:07.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36582025-03-17T16:49:07.257ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
36592025-03-17T16:49:07.257ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36602025-03-17T16:49:07.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36612025-03-17T16:49:07.268ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
36622025-03-17T16:49:07.268ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce23ea9c-6083-4117-97a7-ff99baa8d400", "content-length": "134", "date": "Mon, 17 Mar 2025 16:49:06 GMT"} })
36632025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
36642025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
36652025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
36662025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
36672025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36682025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36692025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36702025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36712025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
36722025-03-17T16:49:07.268ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
36732025-03-17T16:49:07.269ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
36742025-03-17T16:49:07.269ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36752025-03-17T16:49:07.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36762025-03-17T16:49:07.269ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
36772025-03-17T16:49:07.269ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "05f91ad4-863f-4420-b590-2af69c9be944", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
36782025-03-17T16:49:07.269ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
36792025-03-17T16:49:07.270ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b28395f8-c5ce-4740-9202-145f93afc3d5", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
36802025-03-17T16:49:07.270ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
36812025-03-17T16:49:07.271ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
36822025-03-17T16:49:07.271ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d424801-60d6-4a6c-9385-722f7d625448", "content-length": "2", "date": "Mon, 17 Mar 2025 16:49:06 GMT"} })
36832025-03-17T16:49:07.271ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
36842025-03-17T16:49:07.271ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "67cbe854-76d4-48f1-b680-4b0217f27c51", "content-length": "52", "date": "Mon, 17 Mar 2025 16:49:06 GMT"} })
36852025-03-17T16:49:07.271ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
36862025-03-17T16:49:07.271ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
36872025-03-17T16:49:07.272ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "ce183257-bfd4-409b-8db3-b8ce7ebeee28", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
36882025-03-17T16:49:07.272ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
36892025-03-17T16:49:07.274ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
36902025-03-17T16:49:07.274ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 681.648409ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
36912025-03-17T16:49:07.364ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "43ed6852-c5fe-40b4-8100-bcdef2fcea1d", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} })
36922025-03-17T16:49:07.364ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
36932025-03-17T16:49:07.364ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
36942025-03-17T16:49:07.364ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f2c4455-e1e6-4dc6-ae2d-527676da70d9", "content-length": "682", "date": "Mon, 17 Mar 2025 16:49:07 GMT"} })
36952025-03-17T16:49:07.364ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36962025-03-17T16:49:07.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36972025-03-17T16:49:07.369ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36982025-03-17T16:49:07.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36992025-03-17T16:49:07.374ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37002025-03-17T16:49:07.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37012025-03-17T16:49:07.382ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37022025-03-17T16:49:07.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37032025-03-17T16:49:07.427ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 831.608869ms iteration = 17 status = {}
37042025-03-17T16:49:10.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 274 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 35e4bc72-94e6-45c1-a177-dc55e78de52d response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
37052025-03-17T16:49:16.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
37062025-03-17T16:49:16.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37072025-03-17T16:49:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37082025-03-17T16:49:16.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
37092025-03-17T16:49:16.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37102025-03-17T16:49:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37112025-03-17T16:49:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37122025-03-17T16:49:16.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37132025-03-17T16:49:16.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37142025-03-17T16:49:16.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.841204ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
37152025-03-17T16:49:16.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
37162025-03-17T16:49:16.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37172025-03-17T16:49:16.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37182025-03-17T16:49:16.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37192025-03-17T16:49:16.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37202025-03-17T16:49:16.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37212025-03-17T16:49:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
37222025-03-17T16:49:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37232025-03-17T16:49:16.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37242025-03-17T16:49:16.612ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
37252025-03-17T16:49:16.612ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.686632ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
37262025-03-17T16:49:20.124ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 3b3dbc4c-2a98-4bdc-bdbd-cf1d504c8d78 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
37272025-03-17T16:49:20.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 3b3dbc4c-2a98-4bdc-bdbd-cf1d504c8d78 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
37282025-03-17T16:49:20.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3196 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 3b3dbc4c-2a98-4bdc-bdbd-cf1d504c8d78 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
37292025-03-17T16:49:20.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 0a4b5ab3-d18c-413d-9e1a-947f87defce6 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
37302025-03-17T16:49:20.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 0a4b5ab3-d18c-413d-9e1a-947f87defce6 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
37312025-03-17T16:49:20.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2461 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 0a4b5ab3-d18c-413d-9e1a-947f87defce6 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
37322025-03-17T16:49:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 316 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 83642c40-2fb8-490e-939a-99c4d3b7e66d response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
37332025-03-17T16:49:26.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
37342025-03-17T16:49:26.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37352025-03-17T16:49:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37362025-03-17T16:49:26.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
37372025-03-17T16:49:26.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37382025-03-17T16:49:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37392025-03-17T16:49:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37402025-03-17T16:49:26.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37412025-03-17T16:49:26.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37422025-03-17T16:49:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
37432025-03-17T16:49:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37442025-03-17T16:49:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37452025-03-17T16:49:26.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.123389ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
37462025-03-17T16:49:26.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37472025-03-17T16:49:26.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37482025-03-17T16:49:26.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37492025-03-17T16:49:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
37502025-03-17T16:49:26.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37512025-03-17T16:49:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37522025-03-17T16:49:26.611ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
37532025-03-17T16:49:26.611ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.101688ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
37542025-03-17T16:49:30.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 430 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 6085d67e-eff0-4552-a377-b3e5d62db149 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
37552025-03-17T16:49:35.124ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 5a13043f-97f2-46fe-b2ac-4bcf91ea8170 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
37562025-03-17T16:49:35.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 5a13043f-97f2-46fe-b2ac-4bcf91ea8170 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
37572025-03-17T16:49:35.126ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2392 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = 5a13043f-97f2-46fe-b2ac-4bcf91ea8170 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
37582025-03-17T16:49:35.127ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = c7ccedb4-ed86-4014-90b3-570567bb3db7 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
37592025-03-17T16:49:35.127ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = c7ccedb4-ed86-4014-90b3-570567bb3db7 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
37602025-03-17T16:49:35.129ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2321 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:39341 req_id = c7ccedb4-ed86-4014-90b3-570567bb3db7 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
37612025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
37622025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
37632025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
37642025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
37652025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37662025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
37672025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
37682025-03-17T16:49:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
37692025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
37702025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37712025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
37722025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
37732025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37742025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
37752025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37762025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
37772025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37782025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
37792025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37802025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
37812025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
37822025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
37832025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
37842025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37852025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
37862025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
37872025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
37882025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37892025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37902025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37912025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
37922025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37932025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
37942025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
37952025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37962025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
37972025-03-17T16:49:36.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
37982025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37992025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38002025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38012025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38022025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
38032025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38042025-03-17T16:49:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
38052025-03-17T16:49:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
38062025-03-17T16:49:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38072025-03-17T16:49:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
38082025-03-17T16:49:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38092025-03-17T16:49:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38102025-03-17T16:49:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38112025-03-17T16:49:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38122025-03-17T16:49:36.596ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.671197ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
38132025-03-17T16:49:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4263, tv_nsec: 858522905 } })
38142025-03-17T16:49:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4263, tv_nsec: 858678178 } })
38152025-03-17T16:49:36.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
38162025-03-17T16:49:36.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
38172025-03-17T16:49:36.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38182025-03-17T16:49:36.597ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38192025-03-17T16:49:36.598ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38202025-03-17T16:49:36.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 4.165459ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
38212025-03-17T16:49:36.599ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
38222025-03-17T16:49:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38232025-03-17T16:49:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
38242025-03-17T16:49:36.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38252025-03-17T16:49:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38262025-03-17T16:49:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
38272025-03-17T16:49:36.601ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38282025-03-17T16:49:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38292025-03-17T16:49:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
38302025-03-17T16:49:36.601ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38312025-03-17T16:49:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38322025-03-17T16:49:36.602ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38332025-03-17T16:49:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38342025-03-17T16:49:36.602ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38352025-03-17T16:49:36.604ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38362025-03-17T16:49:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
38372025-03-17T16:49:36.604ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
38382025-03-17T16:49:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
38392025-03-17T16:49:36.606ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
38402025-03-17T16:49:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
38412025-03-17T16:49:36.606ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
38422025-03-17T16:49:36.606ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38432025-03-17T16:49:36.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38442025-03-17T16:49:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38452025-03-17T16:49:36.607ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38462025-03-17T16:49:36.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38472025-03-17T16:49:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38482025-03-17T16:49:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
38492025-03-17T16:49:36.609ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38502025-03-17T16:49:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38512025-03-17T16:49:36.609ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38522025-03-17T16:49:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38532025-03-17T16:49:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38542025-03-17T16:49:36.611ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38552025-03-17T16:49:36.611ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38562025-03-17T16:49:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38572025-03-17T16:49:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38582025-03-17T16:49:36.612ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 18.365023ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
38592025-03-17T16:49:36.613ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38602025-03-17T16:49:36.613ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 18.757722ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
38612025-03-17T16:49:36.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38622025-03-17T16:49:36.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38632025-03-17T16:49:36.614ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38642025-03-17T16:49:36.615ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38652025-03-17T16:49:36.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
38662025-03-17T16:49:36.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
38672025-03-17T16:49:36.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
38682025-03-17T16:49:36.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
38692025-03-17T16:49:36.615ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4263, tv_nsec: 878057963 } })
38702025-03-17T16:49:36.616ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
38712025-03-17T16:49:36.616ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38722025-03-17T16:49:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
38732025-03-17T16:49:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
38742025-03-17T16:49:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38752025-03-17T16:49:36.616ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
38762025-03-17T16:49:36.616ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38772025-03-17T16:49:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38782025-03-17T16:49:36.617ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
38792025-03-17T16:49:36.617ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 23.384381ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
38802025-03-17T16:49:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
38812025-03-17T16:49:36.618ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38822025-03-17T16:49:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38832025-03-17T16:49:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
38842025-03-17T16:49:36.618ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38852025-03-17T16:49:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38862025-03-17T16:49:36.620ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 5.176881ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
38872025-03-17T16:49:36.621ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38882025-03-17T16:49:36.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38892025-03-17T16:49:36.625ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38902025-03-17T16:49:36.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38912025-03-17T16:49:36.638ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38922025-03-17T16:49:36.639ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38932025-03-17T16:49:36.639ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
38942025-03-17T16:49:36.639ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38952025-03-17T16:49:36.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38962025-03-17T16:49:36.639ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
38972025-03-17T16:49:36.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38982025-03-17T16:49:36.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
38992025-03-17T16:49:36.644ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39002025-03-17T16:49:36.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39012025-03-17T16:49:36.645ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39022025-03-17T16:49:36.645ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39032025-03-17T16:49:36.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39042025-03-17T16:49:36.646ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
39052025-03-17T16:49:36.646ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.866122ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
39062025-03-17T16:49:36.648ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 53.877745ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
39072025-03-17T16:49:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
39082025-03-17T16:49:36.649ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39092025-03-17T16:49:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39102025-03-17T16:49:36.651ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 57.224766ms iteration = 18 status = 0
39112025-03-17T16:49:36.660ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 44.429821ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
39122025-03-17T16:49:36.662ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 67.883864ms iteration = 19 status = {"physical_disks_added":0}
39132025-03-17T16:49:36.667ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
39142025-03-17T16:49:36.667ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39152025-03-17T16:49:36.667ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "86b27155-f658-437b-9700-bd5529540919", "content-length": "26", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39162025-03-17T16:49:36.668ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39172025-03-17T16:49:36.669ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39182025-03-17T16:49:36.670ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
39192025-03-17T16:49:36.670ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39202025-03-17T16:49:36.670ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "476a0258-e056-4e53-8ea6-15d9cba4a1a4", "content-length": "26", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39212025-03-17T16:49:36.670ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39222025-03-17T16:49:36.671ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
39232025-03-17T16:49:36.671ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 77.230504ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
39242025-03-17T16:49:36.673ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
39252025-03-17T16:49:36.675ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39262025-03-17T16:49:36.675ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
39272025-03-17T16:49:36.675ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
39282025-03-17T16:49:36.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
39292025-03-17T16:49:36.695ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 101.002574ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
39302025-03-17T16:49:36.700ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
39312025-03-17T16:49:36.701ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
39322025-03-17T16:49:36.701ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdce80b4-6975-4a26-a9b8-e6d4da9417ec", "content-length": "26", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39332025-03-17T16:49:36.701ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
39342025-03-17T16:49:36.703ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
39352025-03-17T16:49:36.711ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39362025-03-17T16:49:36.711ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
39372025-03-17T16:49:36.711ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 117.114818ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
39382025-03-17T16:49:36.722ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39392025-03-17T16:49:36.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39402025-03-17T16:49:36.766ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
39412025-03-17T16:49:36.766ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcd9ec15-ddb4-4701-85df-0d2e90e5fcef", "content-length": "1", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39422025-03-17T16:49:36.768ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
39432025-03-17T16:49:36.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
39442025-03-17T16:49:36.771ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 178.505233ms iteration = 18 status = {}
39452025-03-17T16:49:36.777ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
39462025-03-17T16:49:36.777ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
39472025-03-17T16:49:36.778ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f692c0fc-3407-4bb0-926c-028538f5eb56", "content-length": "372", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39482025-03-17T16:49:36.778ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39492025-03-17T16:49:36.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39502025-03-17T16:49:36.779ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39512025-03-17T16:49:36.780ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 187.351772ms iteration = 18 status = null
39522025-03-17T16:49:36.781ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39532025-03-17T16:49:36.781ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39542025-03-17T16:49:36.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39552025-03-17T16:49:36.782ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39562025-03-17T16:49:36.784ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39572025-03-17T16:49:36.784ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39582025-03-17T16:49:36.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39592025-03-17T16:49:36.826ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
39602025-03-17T16:49:36.826ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
39612025-03-17T16:49:36.830ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "82be217f-a138-4a5a-ad61-546f49b9a6b8", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39622025-03-17T16:49:36.830ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39632025-03-17T16:49:36.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39642025-03-17T16:49:36.833ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39652025-03-17T16:49:36.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39662025-03-17T16:49:36.836ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39672025-03-17T16:49:36.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39682025-03-17T16:49:36.839ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
39692025-03-17T16:49:36.839ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
39702025-03-17T16:49:36.840ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "e5d18121-4147-4d9e-a139-44a4457b84f8", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39712025-03-17T16:49:36.841ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 246.917898ms iteration = 18 status = {}
39722025-03-17T16:49:36.842ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39732025-03-17T16:49:36.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39742025-03-17T16:49:36.844ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
39752025-03-17T16:49:36.844ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39762025-03-17T16:49:36.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39772025-03-17T16:49:36.876ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
39782025-03-17T16:49:36.876ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9149c7a-5d2a-4d6e-a30d-9e128a8bf077", "content-length": "691", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39792025-03-17T16:49:36.876ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
39802025-03-17T16:49:36.877ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
39812025-03-17T16:49:36.877ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
39822025-03-17T16:49:36.877ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 283.236326ms iteration = 18 status = {}
39832025-03-17T16:49:36.887ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
39842025-03-17T16:49:36.887ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b642238-da5c-4f5d-8b4b-791dedac9b5f", "content-length": "134", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
39852025-03-17T16:49:36.887ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
39862025-03-17T16:49:36.887ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
39872025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
39882025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
39892025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39902025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39912025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39922025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39932025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39942025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39952025-03-17T16:49:36.888ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
39962025-03-17T16:49:36.888ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39972025-03-17T16:49:36.888ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39982025-03-17T16:49:36.890ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
39992025-03-17T16:49:36.890ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "832b7718-596a-4123-810d-a5428653530f", "content-length": "2", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
40002025-03-17T16:49:36.890ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
40012025-03-17T16:49:36.890ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ccb6bb-98d5-4d3e-b03d-a2b005496cbe", "content-length": "52", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
40022025-03-17T16:49:36.890ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
40032025-03-17T16:49:36.890ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
40042025-03-17T16:49:36.984ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "ed34e581-43f6-4728-ab2e-38dbae1d66eb", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
40052025-03-17T16:49:36.984ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
40062025-03-17T16:49:36.984ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
40072025-03-17T16:49:36.984ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d16689e5-5e82-42c7-9073-c7445044f6a9", "content-length": "682", "date": "Mon, 17 Mar 2025 16:49:36 GMT"} })
40082025-03-17T16:49:36.984ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40092025-03-17T16:49:36.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40102025-03-17T16:49:36.988ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40112025-03-17T16:49:36.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40122025-03-17T16:49:36.992ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40132025-03-17T16:49:36.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40142025-03-17T16:49:36.994ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40152025-03-17T16:49:36.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40162025-03-17T16:49:37.000ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 405.929034ms iteration = 18 status = {}
40172025-03-17T16:49:37.570ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
40182025-03-17T16:49:37.570ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
40192025-03-17T16:49:37.601ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
40202025-03-17T16:49:37.601ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:61537
40212025-03-17T16:49:37.601ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:61537 req_id = 52a28b96-92aa-41e3-bef7-394b5f2c335b roles = RoleSet { roles: {} } uri = /metrics/producers
40222025-03-17T16:49:37.601ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:61537 req_id = 52a28b96-92aa-41e3-bef7-394b5f2c335b resource = Database result = Ok(()) uri = /metrics/producers
40232025-03-17T16:49:37.640ZINFOnexus: assigned collector to new producer collector_id = 1374f180-03c9-4b69-abf3-a0c92428bb9a file = nexus/src/app/oximeter.rs:76 producer_id = 807f84c2-ce3d-4190-8b1c-85a3b322601f
40242025-03-17T16:49:37.640ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 39232 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:61537 req_id = 52a28b96-92aa-41e3-bef7-394b5f2c335b response_code = 201 uri = /metrics/producers
40252025-03-17T16:49:37.640ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "52a28b96-92aa-41e3-bef7-394b5f2c335b", "content-length": "41", "date": "Mon, 17 Mar 2025 16:49:37 GMT"} })
40262025-03-17T16:49:37.641ZDEBGnexus (producer-registration-task): registered with nexus successfully
40272025-03-17T16:49:37.641ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
40282025-03-17T16:49:40.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 467 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 98836666-f599-4ce6-b45a-a7ea7531475a response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
40292025-03-17T16:49:46.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
40302025-03-17T16:49:46.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40312025-03-17T16:49:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40322025-03-17T16:49:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
40332025-03-17T16:49:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40342025-03-17T16:49:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40352025-03-17T16:49:46.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40362025-03-17T16:49:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40372025-03-17T16:49:46.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40382025-03-17T16:49:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40392025-03-17T16:49:46.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40402025-03-17T16:49:46.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40412025-03-17T16:49:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40422025-03-17T16:49:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
40432025-03-17T16:49:46.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40442025-03-17T16:49:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40452025-03-17T16:49:46.613ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
40462025-03-17T16:49:46.613ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.501519ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
40472025-03-17T16:49:46.622ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40482025-03-17T16:49:46.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40492025-03-17T16:49:46.651ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 57.683066ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40502025-03-17T16:49:50.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 409 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = a41615f1-11e3-4e21-942f-61847a605687 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
40512025-03-17T16:49:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
40522025-03-17T16:49:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40532025-03-17T16:49:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40542025-03-17T16:49:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
40552025-03-17T16:49:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40562025-03-17T16:49:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40572025-03-17T16:49:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40582025-03-17T16:49:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40592025-03-17T16:49:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40602025-03-17T16:49:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40612025-03-17T16:49:56.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40622025-03-17T16:49:56.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40632025-03-17T16:49:56.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40642025-03-17T16:49:56.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
40652025-03-17T16:49:56.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40662025-03-17T16:49:56.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40672025-03-17T16:49:56.613ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
40682025-03-17T16:49:56.613ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.369336ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
40692025-03-17T16:49:56.624ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40702025-03-17T16:49:56.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40712025-03-17T16:49:56.626ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.418295ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40722025-03-17T16:50:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = d748ff84-6d74-43b0-ab17-6e7cd21eb058 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
40732025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
40742025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
40752025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
40762025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
40772025-03-17T16:50:06.593ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:40:06.593370288 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
40782025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
40792025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
40802025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
40812025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
40822025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
40832025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
40842025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
40852025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
40862025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
40872025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
40882025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
40892025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40902025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
40912025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
40922025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
40932025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40942025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
40952025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
40962025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
40972025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
40982025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40992025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
41002025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
41012025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41022025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
41032025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41042025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
41052025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41062025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
41072025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
41082025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
41092025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41102025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
41112025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
41122025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41132025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
41142025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
41152025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
41162025-03-17T16:50:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
41172025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41182025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
41192025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41202025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
41212025-03-17T16:50:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
41222025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
41232025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
41242025-03-17T16:50:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
41252025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.83\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
41262025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
41272025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
41282025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
41292025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
41302025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
41312025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
41322025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
41332025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
41342025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
41352025-03-17T16:50:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
41362025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41372025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41382025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
41392025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
41402025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
41412025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41422025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
41432025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
41442025-03-17T16:50:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4293, tv_nsec: 856436628 } })
41452025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41462025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41472025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41482025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41492025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41502025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
41512025-03-17T16:50:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4293, tv_nsec: 856625942 } })
41522025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
41532025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
41542025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41552025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41562025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
41572025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41582025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41592025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41602025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41612025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41622025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
41632025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
41642025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41652025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41662025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41672025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
41682025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.086363ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
41692025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41702025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41712025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
41722025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.4042ms iteration = 10 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
41732025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41742025-03-17T16:50:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
41752025-03-17T16:50:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
41762025-03-17T16:50:06.595ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41772025-03-17T16:50:06.595ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
41782025-03-17T16:50:06.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
41792025-03-17T16:50:06.596ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:40:06.593370288 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
41802025-03-17T16:50:06.596ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.442603ms iteration = 10 status = {"expiration":"2025-03-17T16:40:06.593370288Z","pruned":[]}
41812025-03-17T16:50:06.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4293, tv_nsec: 859511834 } })
41822025-03-17T16:50:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
41832025-03-17T16:50:06.597ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
41842025-03-17T16:50:06.597ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
41852025-03-17T16:50:06.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41862025-03-17T16:50:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
41872025-03-17T16:50:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.232721ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
41882025-03-17T16:50:06.598ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
41892025-03-17T16:50:06.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
41902025-03-17T16:50:06.599ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4293, tv_nsec: 861749662 } })
41912025-03-17T16:50:06.599ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41922025-03-17T16:50:06.599ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
41932025-03-17T16:50:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41942025-03-17T16:50:06.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41952025-03-17T16:50:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41962025-03-17T16:50:06.602ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
41972025-03-17T16:50:06.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41982025-03-17T16:50:06.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41992025-03-17T16:50:06.604ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42002025-03-17T16:50:06.604ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42012025-03-17T16:50:06.605ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 11.382834ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
42022025-03-17T16:50:06.605ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 11.891545ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
42032025-03-17T16:50:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
42042025-03-17T16:50:06.606ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42052025-03-17T16:50:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42062025-03-17T16:50:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
42072025-03-17T16:50:06.607ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42082025-03-17T16:50:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42092025-03-17T16:50:06.608ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42102025-03-17T16:50:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
42112025-03-17T16:50:06.611ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
42122025-03-17T16:50:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
42132025-03-17T16:50:06.614ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42142025-03-17T16:50:06.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
42152025-03-17T16:50:06.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
42162025-03-17T16:50:06.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
42172025-03-17T16:50:06.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
42182025-03-17T16:50:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
42192025-03-17T16:50:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
42202025-03-17T16:50:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
42212025-03-17T16:50:06.617ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
42222025-03-17T16:50:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
42232025-03-17T16:50:06.626ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42242025-03-17T16:50:06.627ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
42252025-03-17T16:50:06.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
42262025-03-17T16:50:06.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
42272025-03-17T16:50:06.639ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42282025-03-17T16:50:06.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42292025-03-17T16:50:06.649ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42302025-03-17T16:50:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42312025-03-17T16:50:06.652ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42322025-03-17T16:50:06.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42332025-03-17T16:50:06.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42342025-03-17T16:50:06.658ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
42352025-03-17T16:50:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
42362025-03-17T16:50:06.659ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42372025-03-17T16:50:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42382025-03-17T16:50:06.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
42392025-03-17T16:50:06.660ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42402025-03-17T16:50:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42412025-03-17T16:50:06.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42422025-03-17T16:50:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42432025-03-17T16:50:06.664ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4293, tv_nsec: 927034020 } })
42442025-03-17T16:50:06.665ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
42452025-03-17T16:50:06.669ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
42462025-03-17T16:50:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
42472025-03-17T16:50:06.669ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42482025-03-17T16:50:06.670ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42492025-03-17T16:50:06.670ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
42502025-03-17T16:50:06.670ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
42512025-03-17T16:50:06.670ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
42522025-03-17T16:50:06.670ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
42532025-03-17T16:50:06.670ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42542025-03-17T16:50:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
42552025-03-17T16:50:06.672ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
42562025-03-17T16:50:06.672ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
42572025-03-17T16:50:06.672ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 78.875799ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
42582025-03-17T16:50:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
42592025-03-17T16:50:06.673ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42602025-03-17T16:50:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42612025-03-17T16:50:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
42622025-03-17T16:50:06.673ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
42632025-03-17T16:50:06.674ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
42642025-03-17T16:50:06.674ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
42652025-03-17T16:50:06.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
42662025-03-17T16:50:06.674ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42672025-03-17T16:50:06.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
42682025-03-17T16:50:06.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42692025-03-17T16:50:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
42702025-03-17T16:50:06.676ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
42712025-03-17T16:50:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
42722025-03-17T16:50:06.676ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42732025-03-17T16:50:06.678ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42742025-03-17T16:50:06.678ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
42752025-03-17T16:50:06.678ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
42762025-03-17T16:50:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42772025-03-17T16:50:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
42782025-03-17T16:50:06.679ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
42792025-03-17T16:50:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
42802025-03-17T16:50:06.679ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42812025-03-17T16:50:06.680ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42822025-03-17T16:50:06.680ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
42832025-03-17T16:50:06.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
42842025-03-17T16:50:06.681ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
42852025-03-17T16:50:06.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
42862025-03-17T16:50:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
42872025-03-17T16:50:06.683ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
42882025-03-17T16:50:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
42892025-03-17T16:50:06.685ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42902025-03-17T16:50:06.686ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42912025-03-17T16:50:06.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
42922025-03-17T16:50:06.688ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42932025-03-17T16:50:06.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42942025-03-17T16:50:06.691ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 97.704112ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
42952025-03-17T16:50:06.692ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
42962025-03-17T16:50:06.692ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
42972025-03-17T16:50:06.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
42982025-03-17T16:50:06.695ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
42992025-03-17T16:50:06.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
43002025-03-17T16:50:06.700ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 107.127665ms iteration = 5 status = {"nsuccess":5}
43012025-03-17T16:50:06.707ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 91.008689ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
43022025-03-17T16:50:06.714ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 98.455509ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
43032025-03-17T16:50:06.716ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43042025-03-17T16:50:06.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43052025-03-17T16:50:06.718ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43062025-03-17T16:50:06.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43072025-03-17T16:50:06.719ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
43082025-03-17T16:50:06.720ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
43092025-03-17T16:50:06.721ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43102025-03-17T16:50:06.726ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
43112025-03-17T16:50:06.726ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43122025-03-17T16:50:06.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43132025-03-17T16:50:06.729ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
43142025-03-17T16:50:06.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
43152025-03-17T16:50:06.729ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
43162025-03-17T16:50:06.729ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
43172025-03-17T16:50:06.729ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
43182025-03-17T16:50:06.730ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43192025-03-17T16:50:06.730ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43202025-03-17T16:50:06.730ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
43212025-03-17T16:50:06.730ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
43222025-03-17T16:50:06.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43232025-03-17T16:50:06.730ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee2271a-569e-4078-bf7c-cc2c4e3c0349", "content-length": "26", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} })
43242025-03-17T16:50:06.730ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
43252025-03-17T16:50:06.731ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
43262025-03-17T16:50:06.731ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 137.183008ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
43272025-03-17T16:50:06.731ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 137.588957ms iteration = 19 status = 0
43282025-03-17T16:50:06.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
43292025-03-17T16:50:06.731ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 138.098967ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43302025-03-17T16:50:06.732ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 138.532097ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
43312025-03-17T16:50:06.732ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43322025-03-17T16:50:06.732ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
43332025-03-17T16:50:06.732ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43342025-03-17T16:50:06.732ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43352025-03-17T16:50:06.733ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
43362025-03-17T16:50:06.733ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43372025-03-17T16:50:06.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43382025-03-17T16:50:06.735ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
43392025-03-17T16:50:06.735ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43402025-03-17T16:50:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43412025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
43422025-03-17T16:50:06.736ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
43432025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 142.269637ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
43442025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
43452025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
43462025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
43472025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
43482025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43492025-03-17T16:50:06.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43502025-03-17T16:50:06.738ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
43512025-03-17T16:50:06.738ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 144.150798ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
43522025-03-17T16:50:06.739ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43532025-03-17T16:50:06.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43542025-03-17T16:50:06.741ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43552025-03-17T16:50:06.741ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
43562025-03-17T16:50:06.741ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 148.189484ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
43572025-03-17T16:50:06.744ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
43582025-03-17T16:50:06.744ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
43592025-03-17T16:50:06.744ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
43602025-03-17T16:50:06.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
43612025-03-17T16:50:06.744ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
43622025-03-17T16:50:06.744ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43632025-03-17T16:50:06.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43642025-03-17T16:50:06.746ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43652025-03-17T16:50:06.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43662025-03-17T16:50:06.747ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
43672025-03-17T16:50:06.747ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
43682025-03-17T16:50:06.747ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43692025-03-17T16:50:06.747ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
43702025-03-17T16:50:06.747ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 153.63261ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
43712025-03-17T16:50:06.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43722025-03-17T16:50:06.747ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
43732025-03-17T16:50:06.747ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
43742025-03-17T16:50:06.748ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f329fa0-6ed4-4d49-931a-23c17fa76a9d", "content-length": "26", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} })
43752025-03-17T16:50:06.748ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
43762025-03-17T16:50:06.749ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43772025-03-17T16:50:06.750ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
43782025-03-17T16:50:06.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
43792025-03-17T16:50:06.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43802025-03-17T16:50:06.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43812025-03-17T16:50:06.754ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
43822025-03-17T16:50:06.754ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
43832025-03-17T16:50:06.754ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
43842025-03-17T16:50:06.754ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 161.212852ms iteration = 10 status = {"generation":1}
43852025-03-17T16:50:06.769ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
43862025-03-17T16:50:06.769ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
43872025-03-17T16:50:06.769ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43882025-03-17T16:50:06.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43892025-03-17T16:50:06.772ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43902025-03-17T16:50:06.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
43912025-03-17T16:50:06.775ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43922025-03-17T16:50:06.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43932025-03-17T16:50:06.779ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
43942025-03-17T16:50:06.779ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
43952025-03-17T16:50:06.779ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
43962025-03-17T16:50:06.779ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 186.079185ms iteration = 10 status = {"generation":2}
43972025-03-17T16:50:06.780ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
43982025-03-17T16:50:06.780ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
43992025-03-17T16:50:06.780ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 186.645848ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
44002025-03-17T16:50:06.785ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
44012025-03-17T16:50:06.785ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 191.646075ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
44022025-03-17T16:50:06.803ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 209.964086ms iteration = 20 status = {"physical_disks_added":0}
44032025-03-17T16:50:06.805ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
44042025-03-17T16:50:06.805ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 211.714525ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
44052025-03-17T16:50:06.814ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44062025-03-17T16:50:06.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
44072025-03-17T16:50:06.816ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44082025-03-17T16:50:06.818ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
44092025-03-17T16:50:06.818ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44102025-03-17T16:50:06.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
44112025-03-17T16:50:06.819ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44122025-03-17T16:50:06.821ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
44132025-03-17T16:50:06.821ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44142025-03-17T16:50:06.822ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
44152025-03-17T16:50:06.833ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
44162025-03-17T16:50:06.833ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
44172025-03-17T16:50:06.833ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd3ed2e3-726a-4d3a-ac52-fcaef5524db5", "content-length": "26", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} })
44182025-03-17T16:50:06.833ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
44192025-03-17T16:50:06.835ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
44202025-03-17T16:50:06.889ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
44212025-03-17T16:50:06.889ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "461bc078-0b4a-4435-a571-d386f23b1864", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
44222025-03-17T16:50:06.889ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
44232025-03-17T16:50:06.890ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "de1bfdd8-6703-4c96-b728-babc53e1234b", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
44242025-03-17T16:50:06.890ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
44252025-03-17T16:50:06.892ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
44262025-03-17T16:50:06.892ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 298.895901ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
44272025-03-17T16:50:06.910ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 316.951988ms iteration = 19 status = {}
44282025-03-17T16:50:06.999ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
44292025-03-17T16:50:06.999ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
44302025-03-17T16:50:07.000ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd1c4281-1a01-41e6-b92f-4409ff663ce1", "content-length": "372", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} })
44312025-03-17T16:50:07.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44322025-03-17T16:50:07.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44332025-03-17T16:50:07.011ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
44342025-03-17T16:50:07.039ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
44352025-03-17T16:50:07.040ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "84eb6932-6e95-46d3-a444-5ce771dd66d5", "content-length": "1", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} })
44362025-03-17T16:50:07.041ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
44372025-03-17T16:50:07.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
44382025-03-17T16:50:07.041ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
44392025-03-17T16:50:07.042ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd6e9132-7e8f-44cd-80bd-cbe27b4cce82", "content-length": "691", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} })
44402025-03-17T16:50:07.042ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
44412025-03-17T16:50:07.042ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
44422025-03-17T16:50:07.042ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
44432025-03-17T16:50:07.043ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 450.539349ms iteration = 19 status = null
44442025-03-17T16:50:07.044ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 450.921878ms iteration = 19 status = {}
44452025-03-17T16:50:07.044ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
44462025-03-17T16:50:07.044ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
44472025-03-17T16:50:07.045ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1bfe69f6-1e71-4935-8cd0-4fb17ebc6e32", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} })
44482025-03-17T16:50:07.045ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44492025-03-17T16:50:07.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44502025-03-17T16:50:07.047ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44512025-03-17T16:50:07.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44522025-03-17T16:50:07.051ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44532025-03-17T16:50:07.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44542025-03-17T16:50:07.054ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
44552025-03-17T16:50:07.054ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
44562025-03-17T16:50:07.055ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "22003b94-91c6-490c-a1c5-dd275d654f9c", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} })
44572025-03-17T16:50:07.057ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 463.739321ms iteration = 19 status = {}
44582025-03-17T16:50:07.095ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
44592025-03-17T16:50:07.096ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "3417c562-b33d-40bf-96d7-2769cfc6c101", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
44602025-03-17T16:50:07.096ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
44612025-03-17T16:50:07.097ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "8af5dc1f-574d-4ab8-bdbd-749271fb97f2", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
44622025-03-17T16:50:07.097ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
44632025-03-17T16:50:07.098ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "2f1c6eee-b0cb-4aaa-96d9-cf71f4c977b0", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
44642025-03-17T16:50:07.098ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
44652025-03-17T16:50:07.100ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
44662025-03-17T16:50:07.100ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 507.135182ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
44672025-03-17T16:50:07.116ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44682025-03-17T16:50:07.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44692025-03-17T16:50:07.119ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
44702025-03-17T16:50:07.119ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44712025-03-17T16:50:07.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44722025-03-17T16:50:07.129ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
44732025-03-17T16:50:07.129ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "34a40703-2960-4568-bd42-56f8ac2f763b", "content-length": "134", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} })
44742025-03-17T16:50:07.129ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
44752025-03-17T16:50:07.129ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
44762025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
44772025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
44782025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44792025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44802025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44812025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
44822025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
44832025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
44842025-03-17T16:50:07.130ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
44852025-03-17T16:50:07.130ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44862025-03-17T16:50:07.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44872025-03-17T16:50:07.132ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
44882025-03-17T16:50:07.132ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5152ee3-0381-40e3-b6f4-4c043d85c046", "content-length": "2", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} })
44892025-03-17T16:50:07.132ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
44902025-03-17T16:50:07.132ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d47c80-dcff-4287-a046-0031bf97fed2", "content-length": "52", "date": "Mon, 17 Mar 2025 16:50:06 GMT"} })
44912025-03-17T16:50:07.132ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
44922025-03-17T16:50:07.132ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
44932025-03-17T16:50:07.227ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "502ebf73-3dc6-425b-b94f-de9cf7a99b9a", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} })
44942025-03-17T16:50:07.227ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
44952025-03-17T16:50:07.227ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
44962025-03-17T16:50:07.230ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "91d46170-5b7f-4f5a-8bab-14dbda258baa", "content-length": "682", "date": "Mon, 17 Mar 2025 16:50:07 GMT"} })
44972025-03-17T16:50:07.230ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44982025-03-17T16:50:07.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44992025-03-17T16:50:07.233ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45002025-03-17T16:50:07.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45012025-03-17T16:50:07.238ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45022025-03-17T16:50:07.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45032025-03-17T16:50:07.240ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45042025-03-17T16:50:07.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
45052025-03-17T16:50:07.247ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 654.17626ms iteration = 19 status = {}
45062025-03-17T16:50:10.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 343 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 8a0aba3b-7a6c-4d60-8907-8e6068686dbc response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
45072025-03-17T16:50:16.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
45082025-03-17T16:50:16.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45092025-03-17T16:50:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45102025-03-17T16:50:16.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
45112025-03-17T16:50:16.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45122025-03-17T16:50:16.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45132025-03-17T16:50:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45142025-03-17T16:50:16.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45152025-03-17T16:50:16.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45162025-03-17T16:50:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
45172025-03-17T16:50:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45182025-03-17T16:50:16.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.104227ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
45192025-03-17T16:50:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45202025-03-17T16:50:16.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45212025-03-17T16:50:16.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45222025-03-17T16:50:16.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45232025-03-17T16:50:16.603ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
45242025-03-17T16:50:16.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45252025-03-17T16:50:16.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45262025-03-17T16:50:16.612ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
45272025-03-17T16:50:16.612ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.892516ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
45282025-03-17T16:50:20.125ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:40049
45292025-03-17T16:50:20.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = ff0c11de-8d4d-416b-bedc-09cdc8c1c47c roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
45302025-03-17T16:50:20.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = ff0c11de-8d4d-416b-bedc-09cdc8c1c47c resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
45312025-03-17T16:50:20.128ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3265 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = ff0c11de-8d4d-416b-bedc-09cdc8c1c47c response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
45322025-03-17T16:50:20.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = 6568493d-d67d-4d3e-84f2-bc1f713aece5 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
45332025-03-17T16:50:20.129ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = 6568493d-d67d-4d3e-84f2-bc1f713aece5 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
45342025-03-17T16:50:20.131ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2582 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = 6568493d-d67d-4d3e-84f2-bc1f713aece5 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
45352025-03-17T16:50:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 1aa9a16a-c1ea-4c87-aacb-9d8bf55480e5 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
45362025-03-17T16:50:26.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
45372025-03-17T16:50:26.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45382025-03-17T16:50:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45392025-03-17T16:50:26.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
45402025-03-17T16:50:26.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45412025-03-17T16:50:26.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45422025-03-17T16:50:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45432025-03-17T16:50:26.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45442025-03-17T16:50:26.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
45452025-03-17T16:50:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
45462025-03-17T16:50:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45472025-03-17T16:50:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45482025-03-17T16:50:26.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45492025-03-17T16:50:26.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45502025-03-17T16:50:26.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45512025-03-17T16:50:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
45522025-03-17T16:50:26.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45532025-03-17T16:50:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45542025-03-17T16:50:26.613ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
45552025-03-17T16:50:26.613ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.603171ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
45562025-03-17T16:50:26.624ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.77981ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
45572025-03-17T16:50:30.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = d171ac97-50dc-44c7-b422-b8d6a9b75109 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
45582025-03-17T16:50:35.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = ddc6f976-0644-459b-8dc3-0bfd1dd98873 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
45592025-03-17T16:50:35.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = ddc6f976-0644-459b-8dc3-0bfd1dd98873 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
45602025-03-17T16:50:35.129ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3871 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = ddc6f976-0644-459b-8dc3-0bfd1dd98873 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
45612025-03-17T16:50:35.129ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = b323771a-7120-4756-ac50-17a87848f8d4 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
45622025-03-17T16:50:35.129ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = b323771a-7120-4756-ac50-17a87848f8d4 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
45632025-03-17T16:50:35.132ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2847 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = b323771a-7120-4756-ac50-17a87848f8d4 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
45642025-03-17T16:50:36.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
45652025-03-17T16:50:36.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
45662025-03-17T16:50:36.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
45672025-03-17T16:50:36.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
45682025-03-17T16:50:36.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45692025-03-17T16:50:36.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
45702025-03-17T16:50:36.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
45712025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
45722025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45732025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
45742025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
45752025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
45762025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45772025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
45782025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
45792025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45802025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45812025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45822025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
45832025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
45842025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
45852025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
45862025-03-17T16:50:36.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
45872025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45882025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
45892025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45902025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
45912025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45922025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
45932025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45942025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
45952025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
45962025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
45972025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45982025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
45992025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
46002025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
46012025-03-17T16:50:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
46022025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46032025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46042025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46052025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46062025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46072025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46082025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46092025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46102025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46112025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
46122025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
46132025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
46142025-03-17T16:50:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
46152025-03-17T16:50:36.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46162025-03-17T16:50:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4323, tv_nsec: 858710955 } })
46172025-03-17T16:50:36.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
46182025-03-17T16:50:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4323, tv_nsec: 859453831 } })
46192025-03-17T16:50:36.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.464176ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
46202025-03-17T16:50:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
46212025-03-17T16:50:36.597ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46222025-03-17T16:50:36.597ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46232025-03-17T16:50:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
46242025-03-17T16:50:36.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46252025-03-17T16:50:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46262025-03-17T16:50:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46272025-03-17T16:50:36.600ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46282025-03-17T16:50:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46292025-03-17T16:50:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
46302025-03-17T16:50:36.601ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46312025-03-17T16:50:36.601ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46322025-03-17T16:50:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46332025-03-17T16:50:36.602ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
46342025-03-17T16:50:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46352025-03-17T16:50:36.602ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46362025-03-17T16:50:36.603ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46372025-03-17T16:50:36.603ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46382025-03-17T16:50:36.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46392025-03-17T16:50:36.604ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46402025-03-17T16:50:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
46412025-03-17T16:50:36.605ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
46422025-03-17T16:50:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46432025-03-17T16:50:36.606ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46442025-03-17T16:50:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
46452025-03-17T16:50:36.607ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46462025-03-17T16:50:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46472025-03-17T16:50:36.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46482025-03-17T16:50:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46492025-03-17T16:50:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
46502025-03-17T16:50:36.608ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
46512025-03-17T16:50:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
46522025-03-17T16:50:36.609ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46532025-03-17T16:50:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46542025-03-17T16:50:36.610ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46552025-03-17T16:50:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46562025-03-17T16:50:36.611ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46572025-03-17T16:50:36.612ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46582025-03-17T16:50:36.612ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46592025-03-17T16:50:36.612ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4323, tv_nsec: 875026965 } })
46602025-03-17T16:50:36.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46612025-03-17T16:50:36.612ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
46622025-03-17T16:50:36.614ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46632025-03-17T16:50:36.614ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46642025-03-17T16:50:36.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46652025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
46662025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
46672025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
46682025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
46692025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46702025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
46712025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
46722025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
46732025-03-17T16:50:36.616ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46742025-03-17T16:50:36.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46752025-03-17T16:50:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46762025-03-17T16:50:36.618ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46772025-03-17T16:50:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46782025-03-17T16:50:36.625ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 9.912432ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
46792025-03-17T16:50:36.638ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
46802025-03-17T16:50:36.638ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.699278ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
46812025-03-17T16:50:36.639ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46822025-03-17T16:50:36.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46832025-03-17T16:50:36.642ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46842025-03-17T16:50:36.643ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
46852025-03-17T16:50:36.643ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
46862025-03-17T16:50:36.643ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46872025-03-17T16:50:36.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46882025-03-17T16:50:36.645ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
46892025-03-17T16:50:36.645ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46902025-03-17T16:50:36.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46912025-03-17T16:50:36.646ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
46922025-03-17T16:50:36.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46932025-03-17T16:50:36.646ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 52.642697ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
46942025-03-17T16:50:36.648ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46952025-03-17T16:50:36.648ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46962025-03-17T16:50:36.648ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
46972025-03-17T16:50:36.648ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
46982025-03-17T16:50:36.648ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46992025-03-17T16:50:36.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47002025-03-17T16:50:36.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
47012025-03-17T16:50:36.650ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
47022025-03-17T16:50:36.650ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47032025-03-17T16:50:36.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
47042025-03-17T16:50:36.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47052025-03-17T16:50:36.651ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
47062025-03-17T16:50:36.651ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47072025-03-17T16:50:36.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47082025-03-17T16:50:36.653ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 59.528594ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
47092025-03-17T16:50:36.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
47102025-03-17T16:50:36.654ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
47112025-03-17T16:50:36.654ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47122025-03-17T16:50:36.654ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
47132025-03-17T16:50:36.654ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47142025-03-17T16:50:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
47152025-03-17T16:50:36.655ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f675694-2e87-45b9-be75-25cb48d488bf", "content-length": "26", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
47162025-03-17T16:50:36.655ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47172025-03-17T16:50:36.655ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "46aa7c34-ea87-4024-a260-78e15ac2e946", "content-length": "26", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
47182025-03-17T16:50:36.655ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47192025-03-17T16:50:36.656ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 62.241273ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
47202025-03-17T16:50:36.656ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
47212025-03-17T16:50:36.657ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
47222025-03-17T16:50:36.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47232025-03-17T16:50:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47242025-03-17T16:50:36.660ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47252025-03-17T16:50:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47262025-03-17T16:50:36.661ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47272025-03-17T16:50:36.662ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
47282025-03-17T16:50:36.662ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 68.520337ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
47292025-03-17T16:50:36.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
47302025-03-17T16:50:36.664ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47312025-03-17T16:50:36.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47322025-03-17T16:50:36.665ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 49.93701ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
47332025-03-17T16:50:36.672ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
47342025-03-17T16:50:36.672ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47352025-03-17T16:50:36.672ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "669a627a-5719-47ba-9556-4264fd3ce79d", "content-length": "26", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
47362025-03-17T16:50:36.672ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47372025-03-17T16:50:36.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
47382025-03-17T16:50:36.675ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 81.527036ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47392025-03-17T16:50:36.686ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47402025-03-17T16:50:36.686ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
47412025-03-17T16:50:36.686ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 93.083744ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
47422025-03-17T16:50:36.689ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47432025-03-17T16:50:36.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
47442025-03-17T16:50:36.693ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47452025-03-17T16:50:36.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47462025-03-17T16:50:36.693ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 99.45946ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
47472025-03-17T16:50:36.694ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 101.051485ms iteration = 20 status = 0
47482025-03-17T16:50:36.702ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
47492025-03-17T16:50:36.702ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 108.526574ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
47502025-03-17T16:50:36.729ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 135.763968ms iteration = 21 status = {"physical_disks_added":0}
47512025-03-17T16:50:36.762ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 169.607743ms iteration = 20 status = {}
47522025-03-17T16:50:36.766ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
47532025-03-17T16:50:36.766ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfbc20e7-0528-4239-a2e0-59e2d95779c7", "content-length": "1", "date": "Mon, 17 Mar 2025 16:50:35 GMT"} })
47542025-03-17T16:50:36.767ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
47552025-03-17T16:50:36.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
47562025-03-17T16:50:36.770ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 177.629934ms iteration = 20 status = null
47572025-03-17T16:50:36.804ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
47582025-03-17T16:50:36.804ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
47592025-03-17T16:50:36.804ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d0573ff-0e36-4878-8399-fc2904cd99a8", "content-length": "372", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
47602025-03-17T16:50:36.804ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47612025-03-17T16:50:36.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47622025-03-17T16:50:36.808ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
47632025-03-17T16:50:36.809ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "00a9cd15-3bae-4dd7-ae77-b23c7f61bc30", "content-length": "691", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
47642025-03-17T16:50:36.809ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
47652025-03-17T16:50:36.809ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
47662025-03-17T16:50:36.809ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
47672025-03-17T16:50:36.811ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 217.444297ms iteration = 20 status = {}
47682025-03-17T16:50:36.846ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
47692025-03-17T16:50:36.846ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
47702025-03-17T16:50:36.846ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "601b96a3-c779-49ab-844f-8ac643dfbc21", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
47712025-03-17T16:50:36.846ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47722025-03-17T16:50:36.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47732025-03-17T16:50:36.848ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47742025-03-17T16:50:36.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47752025-03-17T16:50:36.852ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
47762025-03-17T16:50:36.852ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
47772025-03-17T16:50:36.855ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47782025-03-17T16:50:36.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47792025-03-17T16:50:36.856ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
47802025-03-17T16:50:36.856ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
47812025-03-17T16:50:36.856ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "8a0c8871-23e5-46cb-aa25-206e73f93449", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
47822025-03-17T16:50:36.857ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
47832025-03-17T16:50:36.857ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47842025-03-17T16:50:36.857ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47852025-03-17T16:50:36.857ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 264.019375ms iteration = 20 status = {}
47862025-03-17T16:50:36.866ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
47872025-03-17T16:50:36.867ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6aee112-1d4e-469c-9dd2-989e25989836", "content-length": "134", "date": "Mon, 17 Mar 2025 16:50:35 GMT"} })
47882025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
47892025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
47902025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
47912025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
47922025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47932025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47942025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47952025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47962025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47972025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47982025-03-17T16:50:36.867ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
47992025-03-17T16:50:36.867ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48002025-03-17T16:50:36.868ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48012025-03-17T16:50:36.869ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
48022025-03-17T16:50:36.870ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "40fa4ebf-6a1d-47cc-9c7e-a6f9f5929e59", "content-length": "2", "date": "Mon, 17 Mar 2025 16:50:35 GMT"} })
48032025-03-17T16:50:36.870ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
48042025-03-17T16:50:36.870ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d28f0154-4cc3-462f-a91b-c7c58a582d4e", "content-length": "52", "date": "Mon, 17 Mar 2025 16:50:35 GMT"} })
48052025-03-17T16:50:36.870ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
48062025-03-17T16:50:36.870ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
48072025-03-17T16:50:36.968ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0cdc93a1-946e-47fa-8aba-ffd09904a821", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
48082025-03-17T16:50:36.968ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
48092025-03-17T16:50:36.968ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
48102025-03-17T16:50:36.968ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb332a72-e0a1-4528-8a4f-3d76813207d3", "content-length": "682", "date": "Mon, 17 Mar 2025 16:50:36 GMT"} })
48112025-03-17T16:50:36.968ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48122025-03-17T16:50:36.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48132025-03-17T16:50:36.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48142025-03-17T16:50:36.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48152025-03-17T16:50:36.975ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48162025-03-17T16:50:36.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48172025-03-17T16:50:36.977ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48182025-03-17T16:50:36.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48192025-03-17T16:50:36.984ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 390.77226ms iteration = 20 status = {}
48202025-03-17T16:50:40.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 324 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 205cb057-ef17-44af-829c-7cab072fa329 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
48212025-03-17T16:50:46.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
48222025-03-17T16:50:46.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48232025-03-17T16:50:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48242025-03-17T16:50:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
48252025-03-17T16:50:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48262025-03-17T16:50:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48272025-03-17T16:50:46.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48282025-03-17T16:50:46.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48292025-03-17T16:50:46.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48302025-03-17T16:50:46.597ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.879803ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
48312025-03-17T16:50:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48322025-03-17T16:50:46.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48332025-03-17T16:50:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48342025-03-17T16:50:46.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48352025-03-17T16:50:46.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48362025-03-17T16:50:46.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48372025-03-17T16:50:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
48382025-03-17T16:50:46.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48392025-03-17T16:50:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48402025-03-17T16:50:46.611ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
48412025-03-17T16:50:46.611ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.22295ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
48422025-03-17T16:50:50.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = abe0b4e3-8a92-4220-8071-b21098566ae1 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
48432025-03-17T16:50:56.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
48442025-03-17T16:50:56.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48452025-03-17T16:50:56.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48462025-03-17T16:50:56.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
48472025-03-17T16:50:56.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48482025-03-17T16:50:56.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48492025-03-17T16:50:56.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48502025-03-17T16:50:56.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48512025-03-17T16:50:56.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48522025-03-17T16:50:56.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48532025-03-17T16:50:56.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48542025-03-17T16:50:56.597ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.001625ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
48552025-03-17T16:50:56.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48562025-03-17T16:50:56.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48572025-03-17T16:50:56.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48582025-03-17T16:50:56.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48592025-03-17T16:50:56.603ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
48602025-03-17T16:50:56.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48612025-03-17T16:50:56.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48622025-03-17T16:50:56.610ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
48632025-03-17T16:50:56.610ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.21608ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
48642025-03-17T16:51:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = e03bbf78-f22c-44d1-8931-5f6e33c20a26 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
48652025-03-17T16:51:05.124ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = a67b95d1-e5ae-4843-a05a-f4f70d2c4107 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
48662025-03-17T16:51:05.124ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = a67b95d1-e5ae-4843-a05a-f4f70d2c4107 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
48672025-03-17T16:51:05.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2798 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = a67b95d1-e5ae-4843-a05a-f4f70d2c4107 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
48682025-03-17T16:51:05.127ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = 5e84ff8c-d914-4839-b2c0-610b446dd5ff roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
48692025-03-17T16:51:05.127ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = 5e84ff8c-d914-4839-b2c0-610b446dd5ff resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
48702025-03-17T16:51:05.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2484 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:40049 req_id = 5e84ff8c-d914-4839-b2c0-610b446dd5ff response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
48712025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
48722025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
48732025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
48742025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
48752025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
48762025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
48772025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
48782025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
48792025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48802025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
48812025-03-17T16:51:06.592ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:41:06.592709289 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
48822025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
48832025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
48842025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
48852025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
48862025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
48872025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
48882025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
48892025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
48902025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
48912025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
48922025-03-17T16:51:06.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
48932025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
48942025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
48952025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
48962025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48972025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
48982025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
48992025-03-17T16:51:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4353, tv_nsec: 855968690 } })
49002025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
49012025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
49022025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
49032025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
49042025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49052025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
49062025-03-17T16:51:06.593ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
49072025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49082025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
49092025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49102025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
49112025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49122025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
49132025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49142025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
49152025-03-17T16:51:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
49162025-03-17T16:51:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4353, tv_nsec: 856175925 } })
49172025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.17\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
49182025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
49192025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
49202025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
49212025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
49222025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49232025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
49242025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
49252025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
49262025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
49272025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49282025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
49292025-03-17T16:51:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
49302025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
49312025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
49322025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
49332025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
49342025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
49352025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
49362025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49372025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
49382025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
49392025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49402025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
49412025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
49422025-03-17T16:51:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
49432025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49442025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
49452025-03-17T16:51:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
49462025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49472025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
49482025-03-17T16:51:06.594ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
49492025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
49502025-03-17T16:51:06.594ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
49512025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
49522025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
49532025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49542025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49552025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49562025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49572025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
49582025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49592025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
49602025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
49612025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
49622025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
49632025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.656556ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
49642025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
49652025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
49662025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49672025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
49682025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49692025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49702025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49712025-03-17T16:51:06.594ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
49722025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49732025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
49742025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.576293ms iteration = 11 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
49752025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49762025-03-17T16:51:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
49772025-03-17T16:51:06.595ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
49782025-03-17T16:51:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
49792025-03-17T16:51:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4353, tv_nsec: 858526315 } })
49802025-03-17T16:51:06.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
49812025-03-17T16:51:06.597ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:41:06.592709289 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
49822025-03-17T16:51:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.573368ms iteration = 11 status = {"expiration":"2025-03-17T16:41:06.592709289Z","pruned":[]}
49832025-03-17T16:51:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.573718ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
49842025-03-17T16:51:06.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
49852025-03-17T16:51:06.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49862025-03-17T16:51:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
49872025-03-17T16:51:06.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4353, tv_nsec: 860632270 } })
49882025-03-17T16:51:06.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
49892025-03-17T16:51:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49902025-03-17T16:51:06.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49912025-03-17T16:51:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49922025-03-17T16:51:06.602ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49932025-03-17T16:51:06.602ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
49942025-03-17T16:51:06.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
49952025-03-17T16:51:06.603ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49962025-03-17T16:51:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49972025-03-17T16:51:06.603ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49982025-03-17T16:51:06.604ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49992025-03-17T16:51:06.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
50002025-03-17T16:51:06.604ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50012025-03-17T16:51:06.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50022025-03-17T16:51:06.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50032025-03-17T16:51:06.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
50042025-03-17T16:51:06.605ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
50052025-03-17T16:51:06.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
50062025-03-17T16:51:06.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50072025-03-17T16:51:06.606ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50082025-03-17T16:51:06.607ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
50092025-03-17T16:51:06.607ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
50102025-03-17T16:51:06.607ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 13.478859ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
50112025-03-17T16:51:06.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
50122025-03-17T16:51:06.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50132025-03-17T16:51:06.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
50142025-03-17T16:51:06.608ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
50152025-03-17T16:51:06.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50162025-03-17T16:51:06.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
50172025-03-17T16:51:06.612ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50182025-03-17T16:51:06.612ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
50192025-03-17T16:51:06.612ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
50202025-03-17T16:51:06.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
50212025-03-17T16:51:06.612ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 18.509677ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
50222025-03-17T16:51:06.612ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50232025-03-17T16:51:06.612ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50242025-03-17T16:51:06.613ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50252025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
50262025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50272025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
50282025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50292025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50302025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50312025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
50322025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
50332025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
50342025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
50352025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
50362025-03-17T16:51:06.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
50372025-03-17T16:51:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
50382025-03-17T16:51:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
50392025-03-17T16:51:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
50402025-03-17T16:51:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50412025-03-17T16:51:06.616ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50422025-03-17T16:51:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50432025-03-17T16:51:06.617ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50442025-03-17T16:51:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
50452025-03-17T16:51:06.619ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50462025-03-17T16:51:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50472025-03-17T16:51:06.625ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50482025-03-17T16:51:06.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50492025-03-17T16:51:06.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50502025-03-17T16:51:06.627ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50512025-03-17T16:51:06.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50522025-03-17T16:51:06.629ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
50532025-03-17T16:51:06.630ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50542025-03-17T16:51:06.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
50552025-03-17T16:51:06.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
50562025-03-17T16:51:06.633ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
50572025-03-17T16:51:06.633ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
50582025-03-17T16:51:06.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
50592025-03-17T16:51:06.638ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4353, tv_nsec: 900498164 } })
50602025-03-17T16:51:06.638ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
50612025-03-17T16:51:06.638ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 45.155407ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
50622025-03-17T16:51:06.639ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(b49948c1-3d97-4990-8121-418e525cfc14, 0), (d538a3b0-8ba5-43e4-800c-5c5fde56cb41, 0), (b4ef1928-0871-4e62-bb1f-ff6174bd62c0, 0), (d2c5a596-b855-4ff8-934a-9400c6884964, 0)] collection_id = b49948c1-3d97-4990-8121-418e525cfc14
50632025-03-17T16:51:06.639ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50642025-03-17T16:51:06.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50652025-03-17T16:51:06.642ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
50662025-03-17T16:51:06.642ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 48.381486ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
50672025-03-17T16:51:06.644ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50682025-03-17T16:51:06.645ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50692025-03-17T16:51:06.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50702025-03-17T16:51:06.646ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
50712025-03-17T16:51:06.646ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
50722025-03-17T16:51:06.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
50732025-03-17T16:51:06.649ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50742025-03-17T16:51:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50752025-03-17T16:51:06.651ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
50762025-03-17T16:51:06.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
50772025-03-17T16:51:06.654ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
50782025-03-17T16:51:06.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
50792025-03-17T16:51:06.654ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
50802025-03-17T16:51:06.654ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 60.976966ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
50812025-03-17T16:51:06.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50822025-03-17T16:51:06.657ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50832025-03-17T16:51:06.659ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
50842025-03-17T16:51:06.659ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50852025-03-17T16:51:06.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50862025-03-17T16:51:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
50872025-03-17T16:51:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50882025-03-17T16:51:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
50892025-03-17T16:51:06.661ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
50902025-03-17T16:51:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
50912025-03-17T16:51:06.662ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
50922025-03-17T16:51:06.662ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
50932025-03-17T16:51:06.662ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
50942025-03-17T16:51:06.662ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
50952025-03-17T16:51:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
50962025-03-17T16:51:06.663ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50972025-03-17T16:51:06.663ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50982025-03-17T16:51:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50992025-03-17T16:51:06.667ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
51002025-03-17T16:51:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
51012025-03-17T16:51:06.669ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
51022025-03-17T16:51:06.669ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
51032025-03-17T16:51:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
51042025-03-17T16:51:06.671ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 77.534431ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
51052025-03-17T16:51:06.671ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
51062025-03-17T16:51:06.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
51072025-03-17T16:51:06.672ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51082025-03-17T16:51:06.672ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51092025-03-17T16:51:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51102025-03-17T16:51:06.673ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 79.503213ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
51112025-03-17T16:51:06.673ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51122025-03-17T16:51:06.674ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
51132025-03-17T16:51:06.674ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
51142025-03-17T16:51:06.674ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
51152025-03-17T16:51:06.674ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
51162025-03-17T16:51:06.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51172025-03-17T16:51:06.676ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
51182025-03-17T16:51:06.676ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51192025-03-17T16:51:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51202025-03-17T16:51:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
51212025-03-17T16:51:06.676ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51222025-03-17T16:51:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51232025-03-17T16:51:06.678ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51242025-03-17T16:51:06.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51252025-03-17T16:51:06.679ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 85.362909ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
51262025-03-17T16:51:06.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51272025-03-17T16:51:06.679ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 85.93698ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51282025-03-17T16:51:06.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51292025-03-17T16:51:06.680ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.693936ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
51302025-03-17T16:51:06.681ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
51312025-03-17T16:51:06.681ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
51322025-03-17T16:51:06.681ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 87.807061ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
51332025-03-17T16:51:06.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51342025-03-17T16:51:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51352025-03-17T16:51:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
51362025-03-17T16:51:06.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
51372025-03-17T16:51:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
51382025-03-17T16:51:06.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
51392025-03-17T16:51:06.684ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
51402025-03-17T16:51:06.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51412025-03-17T16:51:06.686ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 70.483519ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
51422025-03-17T16:51:06.686ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51432025-03-17T16:51:06.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51442025-03-17T16:51:06.687ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
51452025-03-17T16:51:06.687ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
51462025-03-17T16:51:06.687ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
51472025-03-17T16:51:06.687ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51482025-03-17T16:51:06.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
51492025-03-17T16:51:06.689ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51502025-03-17T16:51:06.691ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
51512025-03-17T16:51:06.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
51522025-03-17T16:51:06.691ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51532025-03-17T16:51:06.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
51542025-03-17T16:51:06.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51552025-03-17T16:51:06.692ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
51562025-03-17T16:51:06.692ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51572025-03-17T16:51:06.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51582025-03-17T16:51:06.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51592025-03-17T16:51:06.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51602025-03-17T16:51:06.694ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51612025-03-17T16:51:06.695ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51622025-03-17T16:51:06.695ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51632025-03-17T16:51:06.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51642025-03-17T16:51:06.696ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51652025-03-17T16:51:06.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51662025-03-17T16:51:06.698ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51672025-03-17T16:51:06.698ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51682025-03-17T16:51:06.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51692025-03-17T16:51:06.699ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
51702025-03-17T16:51:06.699ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51712025-03-17T16:51:06.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51722025-03-17T16:51:06.701ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
51732025-03-17T16:51:06.701ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 107.655475ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
51742025-03-17T16:51:06.702ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
51752025-03-17T16:51:06.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
51762025-03-17T16:51:06.702ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51772025-03-17T16:51:06.702ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51782025-03-17T16:51:06.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51792025-03-17T16:51:06.703ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51802025-03-17T16:51:06.705ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
51812025-03-17T16:51:06.705ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51822025-03-17T16:51:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51832025-03-17T16:51:06.706ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51842025-03-17T16:51:06.706ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51852025-03-17T16:51:06.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51862025-03-17T16:51:06.707ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51872025-03-17T16:51:06.709ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51882025-03-17T16:51:06.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51892025-03-17T16:51:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
51902025-03-17T16:51:06.711ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51912025-03-17T16:51:06.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51922025-03-17T16:51:06.713ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
51932025-03-17T16:51:06.713ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 120.718755ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
51942025-03-17T16:51:06.714ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 120.185454ms iteration = 21 status = 0
51952025-03-17T16:51:06.714ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51962025-03-17T16:51:06.717ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
51972025-03-17T16:51:06.717ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
51982025-03-17T16:51:06.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
51992025-03-17T16:51:06.719ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52002025-03-17T16:51:06.722ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
52012025-03-17T16:51:06.722ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
52022025-03-17T16:51:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
52032025-03-17T16:51:06.725ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
52042025-03-17T16:51:06.725ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
52052025-03-17T16:51:06.725ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52062025-03-17T16:51:06.725ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
52072025-03-17T16:51:06.725ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 131.933695ms iteration = 4 status = {"blueprint_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","inventory_collection_id":"d2c5a596-b855-4ff8-934a-9400c6884964","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
52082025-03-17T16:51:06.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
52092025-03-17T16:51:06.733ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52102025-03-17T16:51:06.734ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
52112025-03-17T16:51:06.734ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
52122025-03-17T16:51:06.734ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 140.607941ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
52132025-03-17T16:51:06.735ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
52142025-03-17T16:51:06.735ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
52152025-03-17T16:51:06.735ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52162025-03-17T16:51:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
52172025-03-17T16:51:06.736ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
52182025-03-17T16:51:06.736ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52192025-03-17T16:51:06.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
52202025-03-17T16:51:06.737ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52212025-03-17T16:51:06.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52222025-03-17T16:51:06.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52232025-03-17T16:51:06.739ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
52242025-03-17T16:51:06.739ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
52252025-03-17T16:51:06.739ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
52262025-03-17T16:51:06.739ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 146.704572ms iteration = 11 status = {"generation":2}
52272025-03-17T16:51:06.741ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
52282025-03-17T16:51:06.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52292025-03-17T16:51:06.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
52302025-03-17T16:51:06.741ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
52312025-03-17T16:51:06.743ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = b49948c1-3d97-4990-8121-418e525cfc14 file = nexus/db-queries/src/db/datastore/inventory.rs:1442 ncabooses = 12 nclickhouse_keeper_membership = 0 ncollections = 1 ndatasets = 0 nerrors = 0 nnics = 0 nnvme_disk_firmware = 0 nphysical_disks = 0 nrot_pages = 8 nrots = 2 nsled_agent_zones = 0 nsled_agents = 0 nsps = 2 nzones = 0 nzpools = 0
52322025-03-17T16:51:06.743ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52332025-03-17T16:51:06.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52342025-03-17T16:51:06.744ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
52352025-03-17T16:51:06.744ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
52362025-03-17T16:51:06.744ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
52372025-03-17T16:51:06.744ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 151.696119ms iteration = 11 status = {"generation":1}
52382025-03-17T16:51:06.753ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52392025-03-17T16:51:06.755ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(d538a3b0-8ba5-43e4-800c-5c5fde56cb41, 0), (b4ef1928-0871-4e62-bb1f-ff6174bd62c0, 0), (d2c5a596-b855-4ff8-934a-9400c6884964, 0)]
52402025-03-17T16:51:06.755ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4354, tv_nsec: 18439910 } })
52412025-03-17T16:51:06.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
52422025-03-17T16:51:06.756ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52432025-03-17T16:51:06.757ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
52442025-03-17T16:51:06.768ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
52452025-03-17T16:51:06.768ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 174.504517ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
52462025-03-17T16:51:06.775ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
52472025-03-17T16:51:06.775ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 183.0573ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
52482025-03-17T16:51:06.777ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52492025-03-17T16:51:06.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52502025-03-17T16:51:06.779ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52512025-03-17T16:51:06.781ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
52522025-03-17T16:51:06.781ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52532025-03-17T16:51:06.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52542025-03-17T16:51:06.789ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 195.921016ms iteration = 22 status = {"physical_disks_added":0}
52552025-03-17T16:51:06.809ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52562025-03-17T16:51:06.811ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
52572025-03-17T16:51:06.811ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52582025-03-17T16:51:06.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52592025-03-17T16:51:06.856ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
52602025-03-17T16:51:06.856ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
52612025-03-17T16:51:06.856ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe0a044c-3a26-4778-81c1-8570818de10d", "content-length": "26", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
52622025-03-17T16:51:06.857ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
52632025-03-17T16:51:06.859ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
52642025-03-17T16:51:06.869ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
52652025-03-17T16:51:06.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
52662025-03-17T16:51:06.884ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
52672025-03-17T16:51:06.923ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
52682025-03-17T16:51:06.924ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "18eaa07b-22ed-4243-8365-c8299c6926a2", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52692025-03-17T16:51:06.924ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52702025-03-17T16:51:06.926ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "a442c99b-30aa-4d3c-8920-e9ca8c876d31", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52712025-03-17T16:51:06.926ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52722025-03-17T16:51:06.928ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
52732025-03-17T16:51:06.928ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 335.571277ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
52742025-03-17T16:51:06.959ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
52752025-03-17T16:51:06.959ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
52762025-03-17T16:51:06.959ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "34db8764-ab1a-402c-9d2c-00dceb569184", "content-length": "372", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
52772025-03-17T16:51:06.959ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52782025-03-17T16:51:06.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52792025-03-17T16:51:06.996ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
52802025-03-17T16:51:06.996ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
52812025-03-17T16:51:06.996ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4c19d65-06b3-4a8e-92bc-a45f24e9c05b", "content-length": "26", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
52822025-03-17T16:51:06.996ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
52832025-03-17T16:51:06.998ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
52842025-03-17T16:51:07.028ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
52852025-03-17T16:51:07.028ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
52862025-03-17T16:51:07.029ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c2150ba-566e-420a-a923-74d343044a62", "content-length": "26", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
52872025-03-17T16:51:07.029ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
52882025-03-17T16:51:07.031ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
52892025-03-17T16:51:07.033ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
52902025-03-17T16:51:07.033ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
52912025-03-17T16:51:07.033ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
52922025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
52932025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "139dc906-b6ae-4405-b38d-1bbe763ee6d6", "content-length": "476", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
52942025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
52952025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
52962025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07eb2c9d-bd89-4295-b3f4-9d5ccece0db1", "content-length": "734", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
52972025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
52982025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aa855ba-3463-490d-b72f-51093ed4aa83", "content-length": "111", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
52992025-03-17T16:51:07.034ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
53002025-03-17T16:51:07.035ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40a913f-54e6-4f45-b585-79f5589c41ce", "content-length": "111", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53012025-03-17T16:51:07.035ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
53022025-03-17T16:51:07.035ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1075a319-a9c2-4db1-be00-4343ee188d29", "content-length": "167", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53032025-03-17T16:51:07.035ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
53042025-03-17T16:51:07.035ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a024b878-5c58-4bc6-a11a-4bbefe94d7e9", "content-length": "167", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53052025-03-17T16:51:07.035ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
53062025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b7ea8f1-1939-4970-8948-8c8e7fb26174", "content-length": "175", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53072025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
53082025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40c8317-5238-4b05-86ed-775fd45b0a36", "content-length": "175", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53092025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
53102025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86b06c6e-2018-4bb0-b2b5-fb7c0e326a1b", "content-length": "702", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53112025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
53122025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb7f412e-7bef-4573-af3c-b4133684f98a", "content-length": "727", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53132025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
53142025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f2c6272-e00c-42c5-bb75-ad9242136451", "content-length": "729", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53152025-03-17T16:51:07.036ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
53162025-03-17T16:51:07.037ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e3f1a9f-b824-41b3-9dcb-225fa582093b", "content-length": "728", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53172025-03-17T16:51:07.037ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
53182025-03-17T16:51:07.037ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53192025-03-17T16:51:07.037ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53202025-03-17T16:51:07.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53212025-03-17T16:51:07.039ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53222025-03-17T16:51:07.039ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
53232025-03-17T16:51:07.039ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
53242025-03-17T16:51:07.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53252025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
53262025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57432893-d32d-4f5a-9f0c-68b22e4cbaf0", "content-length": "722", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53272025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
53282025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a8864b-5649-42dc-8274-1b843f4aac6e", "content-length": "109", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53292025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
53302025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "937736a2-f0f2-4734-b9ce-a51ab4d555a6", "content-length": "109", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53312025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
53322025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f8a95c-4618-45d7-94e9-e4b7f2aa1aa8", "content-length": "169", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53332025-03-17T16:51:07.040ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
53342025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c67146d6-323f-4fca-801e-f507bb3164ef", "content-length": "169", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53352025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
53362025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70d18132-aec0-440e-bca0-62a3617f6c31", "content-length": "178", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53372025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
53382025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53392025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbc74d5f-d03f-4c39-a72b-7c68b04ce4be", "content-length": "178", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53402025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
53412025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7a32c24-95cb-4edb-a67e-ef4ec3ecfac8", "content-length": "702", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53422025-03-17T16:51:07.041ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
53432025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53442025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41855cf8-ed4d-43e4-b875-f3f91f57755b", "content-length": "727", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53452025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
53462025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57299cf-8a9e-4665-9e33-9292a5da8d70", "content-length": "729", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53472025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
53482025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "41a7a6c7-558b-4711-9b10-9c2db1a152e8", "content-length": "728", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} })
53492025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53502025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
53512025-03-17T16:51:07.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53522025-03-17T16:51:07.043ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53532025-03-17T16:51:07.043ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "047c6e01-0899-4eed-be69-b5a508016f07", "content-length": "2", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} })
53542025-03-17T16:51:07.043ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
53552025-03-17T16:51:07.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53562025-03-17T16:51:07.044ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53572025-03-17T16:51:07.044ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53582025-03-17T16:51:07.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53592025-03-17T16:51:07.045ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 429.350745ms iteration = 3 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"8007234b-9b03-4ff0-af97-c65524f74b0a","time":"2025-03-17T16:46:07.086835970Z"},{"operation":"list","target_sled":"8007234b-9b03-4ff0-af97-c65524f74b0a","time":"2025-03-17T16:51:07.043280519Z"}]}
53602025-03-17T16:51:07.046ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53612025-03-17T16:51:07.046ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53622025-03-17T16:51:07.046ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53632025-03-17T16:51:07.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53642025-03-17T16:51:07.047ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53652025-03-17T16:51:07.050ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
53662025-03-17T16:51:07.050ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53672025-03-17T16:51:07.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53682025-03-17T16:51:07.050ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53692025-03-17T16:51:07.050ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53702025-03-17T16:51:07.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53712025-03-17T16:51:07.051ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
53722025-03-17T16:51:07.051ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
53732025-03-17T16:51:07.051ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "00580d1a-5441-4fa2-9b94-2923da716eb8", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53742025-03-17T16:51:07.051ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53752025-03-17T16:51:07.051ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53762025-03-17T16:51:07.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53772025-03-17T16:51:07.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53782025-03-17T16:51:07.053ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53792025-03-17T16:51:07.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53802025-03-17T16:51:07.053ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53812025-03-17T16:51:07.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53822025-03-17T16:51:07.056ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53832025-03-17T16:51:07.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53842025-03-17T16:51:07.056ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53852025-03-17T16:51:07.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53862025-03-17T16:51:07.058ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53872025-03-17T16:51:07.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53882025-03-17T16:51:07.059ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
53892025-03-17T16:51:07.059ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
53902025-03-17T16:51:07.059ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ad37093f-6686-4b33-a8ba-d13c659d04e6", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
53912025-03-17T16:51:07.060ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53922025-03-17T16:51:07.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53932025-03-17T16:51:07.061ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53942025-03-17T16:51:07.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
53952025-03-17T16:51:07.061ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 467.904602ms iteration = 21 status = {}
53962025-03-17T16:51:07.062ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53972025-03-17T16:51:07.066ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
53982025-03-17T16:51:07.066ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
53992025-03-17T16:51:07.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54002025-03-17T16:51:07.067ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54012025-03-17T16:51:07.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54022025-03-17T16:51:07.069ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54032025-03-17T16:51:07.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54042025-03-17T16:51:07.070ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54052025-03-17T16:51:07.074ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
54062025-03-17T16:51:07.074ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54072025-03-17T16:51:07.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54082025-03-17T16:51:07.079ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54092025-03-17T16:51:07.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54102025-03-17T16:51:07.081ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54112025-03-17T16:51:07.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54122025-03-17T16:51:07.083ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54132025-03-17T16:51:07.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54142025-03-17T16:51:07.085ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54152025-03-17T16:51:07.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54162025-03-17T16:51:07.085ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
54172025-03-17T16:51:07.087ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54182025-03-17T16:51:07.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54192025-03-17T16:51:07.088ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54202025-03-17T16:51:07.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54212025-03-17T16:51:07.090ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54222025-03-17T16:51:07.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54232025-03-17T16:51:07.092ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54242025-03-17T16:51:07.092ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
54252025-03-17T16:51:07.092ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "aac8f634-b95c-4708-bc48-9541655a939e", "content-length": "1", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
54262025-03-17T16:51:07.094ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
54272025-03-17T16:51:07.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
54282025-03-17T16:51:07.095ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
54292025-03-17T16:51:07.095ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54302025-03-17T16:51:07.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54312025-03-17T16:51:07.096ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 504.25958ms iteration = 21 status = null
54322025-03-17T16:51:07.097ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54332025-03-17T16:51:07.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54342025-03-17T16:51:07.098ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54352025-03-17T16:51:07.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54362025-03-17T16:51:07.099ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54372025-03-17T16:51:07.103ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
54382025-03-17T16:51:07.103ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54392025-03-17T16:51:07.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54402025-03-17T16:51:07.120ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 527.994028ms iteration = 21 status = {}
54412025-03-17T16:51:07.185ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
54422025-03-17T16:51:07.186ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "193cbfaf-4fc5-4ab7-b3e1-c808e4237ad7", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54432025-03-17T16:51:07.186ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54442025-03-17T16:51:07.188ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "203312ee-1188-4751-86d3-77b602609559", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54452025-03-17T16:51:07.188ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54462025-03-17T16:51:07.189ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "61fae1be-464e-4f3c-8948-770abae907df", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54472025-03-17T16:51:07.189ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54482025-03-17T16:51:07.191ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54492025-03-17T16:51:07.191ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 598.746484ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
54502025-03-17T16:51:07.192ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
54512025-03-17T16:51:07.192ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e3a2be2-b009-494d-b83c-8517bfbaec87", "content-length": "691", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
54522025-03-17T16:51:07.192ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
54532025-03-17T16:51:07.192ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
54542025-03-17T16:51:07.192ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
54552025-03-17T16:51:07.193ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 599.98013ms iteration = 21 status = {}
54562025-03-17T16:51:07.194ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 4bbf4949-9681-44f4-9ce1-d34c7cfeaf8e, kind: Service { id: 3827727f-e72b-46db-972f-ef987798c7fe }, name: Name("nexus-3827727f-e72b-46db-972f-ef987798c7fe"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 212, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 64b2110d-a2ad-4bda-9407-3a44fd83a7ef, kind: Service { id: 807f84c2-ce3d-4190-8b1c-85a3b322601f }, name: Name("nexus-807f84c2-ce3d-4190-8b1c-85a3b322601f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c1e71a59-6914-4de0-a06a-4a40418bbcd0, kind: Service { id: b8f58801-e3d7-4a1a-b665-e14ca1b06822 }, name: Name("nexus-b8f58801-e3d7-4a1a-b665-e14ca1b06822"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 130])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 524f6c4e-dd14-4112-b941-145bff1dbf2b, kind: Service { id: 36bd5b8d-4179-4c7d-acfe-dc22937af04b }, name: Name("external-dns-36bd5b8d-4179-4c7d-acfe-dc22937af04b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 238])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 85143d5f-0adb-4664-a2ac-73adcf707df0, kind: Service { id: 91530a96-1dfe-4db2-b8a3-2a09e4675f90 }, name: Name("external-dns-91530a96-1dfe-4db2-b8a3-2a09e4675f90"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 26])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
54572025-03-17T16:51:07.194ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54582025-03-17T16:51:07.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54592025-03-17T16:51:07.195ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54602025-03-17T16:51:07.197ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
54612025-03-17T16:51:07.198ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54622025-03-17T16:51:07.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54632025-03-17T16:51:07.199ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
54642025-03-17T16:51:07.199ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 524f6c4e-dd14-4112-b941-145bff1dbf2b, kind: Service { id: 36bd5b8d-4179-4c7d-acfe-dc22937af04b }, name: Name("external-dns-36bd5b8d-4179-4c7d-acfe-dc22937af04b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 238])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 85143d5f-0adb-4664-a2ac-73adcf707df0, kind: Service { id: 91530a96-1dfe-4db2-b8a3-2a09e4675f90 }, name: Name("external-dns-91530a96-1dfe-4db2-b8a3-2a09e4675f90"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 26])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4bbf4949-9681-44f4-9ce1-d34c7cfeaf8e, kind: Service { id: 3827727f-e72b-46db-972f-ef987798c7fe }, name: Name("nexus-3827727f-e72b-46db-972f-ef987798c7fe"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 212, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 64b2110d-a2ad-4bda-9407-3a44fd83a7ef, kind: Service { id: 807f84c2-ce3d-4190-8b1c-85a3b322601f }, name: Name("nexus-807f84c2-ce3d-4190-8b1c-85a3b322601f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c1e71a59-6914-4de0-a06a-4a40418bbcd0, kind: Service { id: b8f58801-e3d7-4a1a-b665-e14ca1b06822 }, name: Name("nexus-b8f58801-e3d7-4a1a-b665-e14ca1b06822"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 130])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
54652025-03-17T16:51:07.199ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
54662025-03-17T16:51:07.202ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
54672025-03-17T16:51:07.204ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
54682025-03-17T16:51:07.242ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54692025-03-17T16:51:07.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54702025-03-17T16:51:07.245ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
54712025-03-17T16:51:07.245ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54722025-03-17T16:51:07.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54732025-03-17T16:51:07.255ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
54742025-03-17T16:51:07.255ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "086c083f-425a-4ad4-b203-ee2637bb2c0c", "content-length": "134", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
54752025-03-17T16:51:07.255ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
54762025-03-17T16:51:07.255ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
54772025-03-17T16:51:07.255ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
54782025-03-17T16:51:07.255ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
54792025-03-17T16:51:07.256ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54802025-03-17T16:51:07.256ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54812025-03-17T16:51:07.256ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54822025-03-17T16:51:07.256ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
54832025-03-17T16:51:07.256ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54842025-03-17T16:51:07.256ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
54852025-03-17T16:51:07.256ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
54862025-03-17T16:51:07.256ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54872025-03-17T16:51:07.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54882025-03-17T16:51:07.258ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
54892025-03-17T16:51:07.258ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2820f6-3cbf-4254-8a05-d9f17114a071", "content-length": "2", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
54902025-03-17T16:51:07.258ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
54912025-03-17T16:51:07.258ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "0301213e-6df5-4a34-8846-84fd05478d9a", "content-length": "52", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
54922025-03-17T16:51:07.258ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
54932025-03-17T16:51:07.258ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
54942025-03-17T16:51:07.322ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 8007234b-9b03-4ff0-af97-c65524f74b0a, time_created: 2025-03-17T16:41:47.612611Z, time_modified: 2025-03-17T16:41:47.612611Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
54952025-03-17T16:51:07.322ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
54962025-03-17T16:51:07.322ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54972025-03-17T16:51:07.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54982025-03-17T16:51:07.324ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54992025-03-17T16:51:07.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55002025-03-17T16:51:07.326ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55012025-03-17T16:51:07.329ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
55022025-03-17T16:51:07.361ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2193737d-25c2-4735-abdb-2e1df7429fa7", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} })
55032025-03-17T16:51:07.361ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
55042025-03-17T16:51:07.361ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
55052025-03-17T16:51:07.361ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "33166bbb-7a6f-4cb1-a927-662c56f888e3", "content-length": "682", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} })
55062025-03-17T16:51:07.361ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55072025-03-17T16:51:07.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55082025-03-17T16:51:07.362ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
55092025-03-17T16:51:07.363ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "46cd9904-fb7b-4be6-a50f-5f828ada850a", "date": "Mon, 17 Mar 2025 16:51:07 GMT"} })
55102025-03-17T16:51:07.365ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
55112025-03-17T16:51:07.365ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 771.390921ms
55122025-03-17T16:51:07.365ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 771.403022ms iteration = 3 status = {}
55132025-03-17T16:51:07.366ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55142025-03-17T16:51:07.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55152025-03-17T16:51:07.381ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55162025-03-17T16:51:07.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55172025-03-17T16:51:07.384ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55182025-03-17T16:51:07.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55192025-03-17T16:51:07.396ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 803.005098ms iteration = 21 status = {}
55202025-03-17T16:51:07.518ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "08d994d9-12be-4fea-b0ab-e7cf7ca14a64", "content-length": "31973", "date": "Mon, 17 Mar 2025 16:51:06 GMT"} })
55212025-03-17T16:51:07.540ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55222025-03-17T16:51:07.540ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55232025-03-17T16:51:07.540ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
55242025-03-17T16:51:07.544ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55252025-03-17T16:51:07.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55262025-03-17T16:51:07.547ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55272025-03-17T16:51:07.549ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55282025-03-17T16:51:07.549ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55292025-03-17T16:51:07.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55302025-03-17T16:51:07.779ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 06bf1a1a-4d84-4e52-a81c-d863c7e5b472 file = nexus/db-queries/src/db/datastore/inventory.rs:1010
55312025-03-17T16:51:07.779ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 06bf1a1a-4d84-4e52-a81c-d863c7e5b472 time_started = 2025-03-17 16:51:07.033866 UTC
55322025-03-17T16:51:07.779ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
55332025-03-17T16:51:07.779ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.18598113s iteration = 2 status = {"collection_id":"06bf1a1a-4d84-4e52-a81c-d863c7e5b472","time_done":"2025-03-17 16:51:07.542976 UTC","time_started":"2025-03-17 16:51:07.033866 UTC"}
55342025-03-17T16:51:07.779ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
55352025-03-17T16:51:07.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55362025-03-17T16:51:07.779ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55372025-03-17T16:51:07.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55382025-03-17T16:51:07.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55392025-03-17T16:51:07.781ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55402025-03-17T16:51:07.782ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55412025-03-17T16:51:07.785ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
55422025-03-17T16:51:07.785ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55432025-03-17T16:51:07.785ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
55442025-03-17T16:51:07.785ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55452025-03-17T16:51:07.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55462025-03-17T16:51:07.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55472025-03-17T16:51:07.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55482025-03-17T16:51:07.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55492025-03-17T16:51:07.789ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55502025-03-17T16:51:07.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55512025-03-17T16:51:07.791ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55522025-03-17T16:51:07.794ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
55532025-03-17T16:51:07.794ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55542025-03-17T16:51:07.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55552025-03-17T16:51:07.830ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55562025-03-17T16:51:07.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55572025-03-17T16:51:07.860ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55582025-03-17T16:51:07.863ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
55592025-03-17T16:51:07.863ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55602025-03-17T16:51:07.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55612025-03-17T16:51:07.895ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55622025-03-17T16:51:07.896ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 116.642278ms iteration = 23 status = {"physical_disks_added":0}
55632025-03-17T16:51:07.898ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
55642025-03-17T16:51:07.898ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55652025-03-17T16:51:07.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55662025-03-17T16:51:07.901ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55672025-03-17T16:51:07.902ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55682025-03-17T16:51:07.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55692025-03-17T16:51:07.942ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
55702025-03-17T16:51:07.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55712025-03-17T16:51:07.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55722025-03-17T16:51:07.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55732025-03-17T16:51:07.992ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
55742025-03-17T16:51:07.992ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55752025-03-17T16:51:07.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55762025-03-17T16:51:07.995ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
55772025-03-17T16:51:07.995ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 215.627438ms iteration = 5 status = {"blueprint_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","inventory_collection_id":"06bf1a1a-4d84-4e52-a81c-d863c7e5b472","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
55782025-03-17T16:51:10.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 312 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = e4155777-2da9-4a95-b507-5e32318b98e6 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
55792025-03-17T16:51:16.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
55802025-03-17T16:51:16.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55812025-03-17T16:51:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55822025-03-17T16:51:16.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
55832025-03-17T16:51:16.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55842025-03-17T16:51:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55852025-03-17T16:51:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55862025-03-17T16:51:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
55872025-03-17T16:51:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55882025-03-17T16:51:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55892025-03-17T16:51:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55902025-03-17T16:51:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55912025-03-17T16:51:16.628ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55922025-03-17T16:51:16.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55932025-03-17T16:51:16.635ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55942025-03-17T16:51:16.638ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
55952025-03-17T16:51:16.638ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55962025-03-17T16:51:16.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55972025-03-17T16:51:16.648ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
55982025-03-17T16:51:16.648ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 55.268424ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
55992025-03-17T16:51:16.658ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.992051ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
56002025-03-17T16:51:20.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 437 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 2e5425f3-d6fe-462f-9545-cae571e889a3 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
56012025-03-17T16:51:26.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
56022025-03-17T16:51:26.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56032025-03-17T16:51:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56042025-03-17T16:51:26.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
56052025-03-17T16:51:26.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56062025-03-17T16:51:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56072025-03-17T16:51:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56082025-03-17T16:51:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56092025-03-17T16:51:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56102025-03-17T16:51:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56112025-03-17T16:51:26.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56122025-03-17T16:51:26.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56132025-03-17T16:51:26.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56142025-03-17T16:51:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
56152025-03-17T16:51:26.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56162025-03-17T16:51:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56172025-03-17T16:51:26.611ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
56182025-03-17T16:51:26.611ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.791263ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
56192025-03-17T16:51:26.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56202025-03-17T16:51:26.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56212025-03-17T16:51:26.660ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.790951ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
56222025-03-17T16:51:30.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 276 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 9013eeea-208f-45c2-b762-7134183081a4 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
56232025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
56242025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
56252025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
56262025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56272025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
56282025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
56292025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
56302025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56312025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
56322025-03-17T16:51:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
56332025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
56342025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
56352025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
56362025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56372025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56382025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56392025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
56402025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
56412025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56422025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56432025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
56442025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
56452025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
56462025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
56472025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
56482025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
56492025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
56502025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56512025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
56522025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
56532025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
56542025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
56552025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56562025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
56572025-03-17T16:51:36.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
56582025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
56592025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56602025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56612025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56622025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56632025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56642025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
56652025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56662025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
56672025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
56682025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56692025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56702025-03-17T16:51:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56712025-03-17T16:51:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
56722025-03-17T16:51:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56732025-03-17T16:51:36.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56742025-03-17T16:51:36.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56752025-03-17T16:51:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4383, tv_nsec: 858731919 } })
56762025-03-17T16:51:36.596ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.093907ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
56772025-03-17T16:51:36.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
56782025-03-17T16:51:36.596ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56792025-03-17T16:51:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4383, tv_nsec: 859443685 } })
56802025-03-17T16:51:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
56812025-03-17T16:51:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
56822025-03-17T16:51:36.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56832025-03-17T16:51:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56842025-03-17T16:51:36.599ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56852025-03-17T16:51:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56862025-03-17T16:51:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56872025-03-17T16:51:36.602ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
56882025-03-17T16:51:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56892025-03-17T16:51:36.603ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56902025-03-17T16:51:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
56912025-03-17T16:51:36.605ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56922025-03-17T16:51:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56932025-03-17T16:51:36.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56942025-03-17T16:51:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56952025-03-17T16:51:36.607ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56962025-03-17T16:51:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56972025-03-17T16:51:36.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
56982025-03-17T16:51:36.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
56992025-03-17T16:51:36.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
57002025-03-17T16:51:36.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
57012025-03-17T16:51:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
57022025-03-17T16:51:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
57032025-03-17T16:51:36.644ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 49.911869ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
57042025-03-17T16:51:36.644ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57052025-03-17T16:51:36.644ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57062025-03-17T16:51:36.645ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
57072025-03-17T16:51:36.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
57082025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57092025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57102025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
57112025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
57122025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57132025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57142025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57152025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57162025-03-17T16:51:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
57172025-03-17T16:51:36.648ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
57182025-03-17T16:51:36.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
57192025-03-17T16:51:36.649ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
57202025-03-17T16:51:36.649ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57212025-03-17T16:51:36.649ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
57222025-03-17T16:51:36.649ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57232025-03-17T16:51:36.650ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b069ccd6-738e-43cb-8312-917b63a72b97", "content-length": "26", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
57242025-03-17T16:51:36.650ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57252025-03-17T16:51:36.650ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d67bc798-e036-48a4-b505-0bd577abc99c", "content-length": "26", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
57262025-03-17T16:51:36.650ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57272025-03-17T16:51:36.651ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4383, tv_nsec: 913821609 } })
57282025-03-17T16:51:36.651ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57292025-03-17T16:51:36.651ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
57302025-03-17T16:51:36.651ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
57312025-03-17T16:51:36.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
57322025-03-17T16:51:36.652ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
57332025-03-17T16:51:36.652ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
57342025-03-17T16:51:36.652ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57352025-03-17T16:51:36.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57362025-03-17T16:51:36.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57372025-03-17T16:51:36.658ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57382025-03-17T16:51:36.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57392025-03-17T16:51:36.660ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
57402025-03-17T16:51:36.660ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 65.616995ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
57412025-03-17T16:51:36.660ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57422025-03-17T16:51:36.660ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
57432025-03-17T16:51:36.660ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57442025-03-17T16:51:36.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
57452025-03-17T16:51:36.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57462025-03-17T16:51:36.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57472025-03-17T16:51:36.661ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.885352ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57482025-03-17T16:51:36.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57492025-03-17T16:51:36.661ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57502025-03-17T16:51:36.661ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57512025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 47.908786ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
57522025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 47.964067ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
57532025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
57542025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57552025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
57562025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
57572025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57582025-03-17T16:51:36.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57592025-03-17T16:51:36.666ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57602025-03-17T16:51:36.670ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
57612025-03-17T16:51:36.670ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57622025-03-17T16:51:36.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57632025-03-17T16:51:36.675ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 81.412393ms iteration = 24 status = {"physical_disks_added":0}
57642025-03-17T16:51:36.677ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
57652025-03-17T16:51:36.677ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 83.971308ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
57662025-03-17T16:51:36.685ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57672025-03-17T16:51:36.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57682025-03-17T16:51:36.687ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57692025-03-17T16:51:36.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57702025-03-17T16:51:36.688ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57712025-03-17T16:51:36.688ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 94.425092ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
57722025-03-17T16:51:36.691ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57732025-03-17T16:51:36.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57742025-03-17T16:51:36.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
57752025-03-17T16:51:36.691ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57762025-03-17T16:51:36.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57772025-03-17T16:51:36.692ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 98.266705ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57782025-03-17T16:51:36.692ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57792025-03-17T16:51:36.694ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 99.565653ms iteration = 22 status = 0
57802025-03-17T16:51:36.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
57812025-03-17T16:51:36.694ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57822025-03-17T16:51:36.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
57832025-03-17T16:51:36.695ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 101.479964ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57842025-03-17T16:51:36.702ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57852025-03-17T16:51:36.702ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
57862025-03-17T16:51:36.702ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57872025-03-17T16:51:36.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57882025-03-17T16:51:36.724ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
57892025-03-17T16:51:36.724ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 130.189618ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
57902025-03-17T16:51:36.725ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57912025-03-17T16:51:36.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57922025-03-17T16:51:36.729ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
57932025-03-17T16:51:36.729ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
57942025-03-17T16:51:36.729ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
57952025-03-17T16:51:36.729ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b8d77ea-5293-4d90-91c1-021df82d2110", "content-length": "1", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
57962025-03-17T16:51:36.730ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a3fba89-51e3-496e-a1c8-0a8c3fd1ded4", "content-length": "26", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
57972025-03-17T16:51:36.730ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
57982025-03-17T16:51:36.732ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57992025-03-17T16:51:36.732ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
58002025-03-17T16:51:36.732ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 138.153459ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
58012025-03-17T16:51:36.734ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
58022025-03-17T16:51:36.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
58032025-03-17T16:51:36.734ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
58042025-03-17T16:51:36.735ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 141.710385ms iteration = 22 status = {}
58052025-03-17T16:51:36.736ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 143.363191ms iteration = 22 status = null
58062025-03-17T16:51:36.789ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
58072025-03-17T16:51:36.789ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
58082025-03-17T16:51:36.789ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "db66f4b7-9e6a-43c6-bd26-aed2a242ba5a", "content-length": "372", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58092025-03-17T16:51:36.789ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58102025-03-17T16:51:36.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58112025-03-17T16:51:36.804ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
58122025-03-17T16:51:36.804ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
58132025-03-17T16:51:36.805ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a483d2d6-ee83-4acd-892e-a961a5c32708", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58142025-03-17T16:51:36.805ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58152025-03-17T16:51:36.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58162025-03-17T16:51:36.807ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58172025-03-17T16:51:36.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58182025-03-17T16:51:36.810ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58192025-03-17T16:51:36.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58202025-03-17T16:51:36.814ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
58212025-03-17T16:51:36.814ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
58222025-03-17T16:51:36.814ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a8758648-7d31-4784-8ae3-71df68ee9219", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58232025-03-17T16:51:36.816ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 221.826061ms iteration = 22 status = {}
58242025-03-17T16:51:36.818ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58252025-03-17T16:51:36.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58262025-03-17T16:51:36.820ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58272025-03-17T16:51:36.822ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58282025-03-17T16:51:36.822ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58292025-03-17T16:51:36.822ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58302025-03-17T16:51:36.823ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58312025-03-17T16:51:36.825ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
58322025-03-17T16:51:36.825ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
58332025-03-17T16:51:36.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58342025-03-17T16:51:36.854ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58352025-03-17T16:51:36.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58362025-03-17T16:51:36.856ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
58372025-03-17T16:51:36.856ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58382025-03-17T16:51:36.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58392025-03-17T16:51:36.866ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
58402025-03-17T16:51:36.867ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "24d53568-a91e-4d94-b6f0-6e6af9bd6596", "content-length": "134", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58412025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
58422025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
58432025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
58442025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
58452025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58462025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58472025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58482025-03-17T16:51:36.867ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58492025-03-17T16:51:36.868ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
58502025-03-17T16:51:36.868ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
58512025-03-17T16:51:36.868ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
58522025-03-17T16:51:36.868ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58532025-03-17T16:51:36.868ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58542025-03-17T16:51:36.872ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
58552025-03-17T16:51:36.873ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "10bbc30e-4deb-4fd7-a961-5b8817143374", "content-length": "2", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58562025-03-17T16:51:36.873ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
58572025-03-17T16:51:36.874ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cdc277a-81a4-4012-b726-e8f26d505724", "content-length": "52", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58582025-03-17T16:51:36.874ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
58592025-03-17T16:51:36.874ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
58602025-03-17T16:51:36.919ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
58612025-03-17T16:51:36.962ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "910681c9-ba49-4c82-81c2-2f0f1626d61e", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58622025-03-17T16:51:36.962ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
58632025-03-17T16:51:36.962ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
58642025-03-17T16:51:36.963ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb4083a0-edd6-4ac4-a317-8a58b2bfd43c", "content-length": "691", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58652025-03-17T16:51:36.963ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
58662025-03-17T16:51:36.963ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
58672025-03-17T16:51:36.963ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
58682025-03-17T16:51:36.963ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "983195a2-897c-423e-a430-bedfd6eb74a8", "content-length": "682", "date": "Mon, 17 Mar 2025 16:51:36 GMT"} })
58692025-03-17T16:51:36.963ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58702025-03-17T16:51:36.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58712025-03-17T16:51:36.964ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 369.668477ms iteration = 22 status = {}
58722025-03-17T16:51:36.967ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58732025-03-17T16:51:36.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58742025-03-17T16:51:36.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58752025-03-17T16:51:36.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58762025-03-17T16:51:36.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58772025-03-17T16:51:36.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58782025-03-17T16:51:36.978ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 384.501594ms iteration = 22 status = {}
58792025-03-17T16:51:40.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 351 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = d929e1ee-89dc-4f89-a86c-446704f43f8c response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
58802025-03-17T16:51:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
58812025-03-17T16:51:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58822025-03-17T16:51:46.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
58832025-03-17T16:51:46.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58842025-03-17T16:51:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58852025-03-17T16:51:46.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58862025-03-17T16:51:46.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58872025-03-17T16:51:46.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58882025-03-17T16:51:46.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58892025-03-17T16:51:46.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58902025-03-17T16:51:46.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58912025-03-17T16:51:46.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58922025-03-17T16:51:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58932025-03-17T16:51:46.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58942025-03-17T16:51:46.627ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.809064ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
58952025-03-17T16:51:46.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58962025-03-17T16:51:46.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
58972025-03-17T16:51:46.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58982025-03-17T16:51:46.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58992025-03-17T16:51:46.642ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
59002025-03-17T16:51:46.642ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 48.667382ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
59012025-03-17T16:51:50.125ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:60809
59022025-03-17T16:51:50.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60809 req_id = dc1c20b3-d3e5-48cc-8c75-aae7ffbf81d0 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
59032025-03-17T16:51:50.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60809 req_id = dc1c20b3-d3e5-48cc-8c75-aae7ffbf81d0 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
59042025-03-17T16:51:50.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2575 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60809 req_id = dc1c20b3-d3e5-48cc-8c75-aae7ffbf81d0 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
59052025-03-17T16:51:50.127ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60809 req_id = aa3df6f0-9aba-4e03-83da-43ba935b2983 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
59062025-03-17T16:51:50.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60809 req_id = aa3df6f0-9aba-4e03-83da-43ba935b2983 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
59072025-03-17T16:51:50.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2483 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60809 req_id = aa3df6f0-9aba-4e03-83da-43ba935b2983 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
59082025-03-17T16:51:50.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 264 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 653d841d-1341-4afa-b8b0-324fdf0e60df response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
59092025-03-17T16:51:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
59102025-03-17T16:51:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59112025-03-17T16:51:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59122025-03-17T16:51:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
59132025-03-17T16:51:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59142025-03-17T16:51:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59152025-03-17T16:51:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59162025-03-17T16:51:56.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59172025-03-17T16:51:56.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59182025-03-17T16:51:56.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.800362ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59192025-03-17T16:51:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59202025-03-17T16:51:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59212025-03-17T16:51:56.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59222025-03-17T16:51:56.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59232025-03-17T16:51:56.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59242025-03-17T16:51:56.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59252025-03-17T16:51:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
59262025-03-17T16:51:56.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59272025-03-17T16:51:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59282025-03-17T16:51:56.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
59292025-03-17T16:51:56.614ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.333927ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
59302025-03-17T16:52:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 324 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 894a433f-1e01-4387-93ad-cf140baf49ee response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
59312025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
59322025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
59332025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
59342025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
59352025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
59362025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59372025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
59382025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
59392025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
59402025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
59412025-03-17T16:52:06.593ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:42:06.593378871 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
59422025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
59432025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
59442025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
59452025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
59462025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
59472025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
59482025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
59492025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
59502025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59512025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
59522025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
59532025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
59542025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
59552025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
59562025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59572025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
59582025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
59592025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59602025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
59612025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
59622025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
59632025-03-17T16:52:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
59642025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59652025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59662025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
59672025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59682025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
59692025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
59702025-03-17T16:52:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
59712025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.23\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
59722025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
59732025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
59742025-03-17T16:52:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
59752025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
59762025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
59772025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
59782025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
59792025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
59802025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
59812025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
59822025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
59832025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
59842025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59852025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
59862025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
59872025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
59882025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59892025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
59902025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59912025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
59922025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
59932025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59942025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
59952025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
59962025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
59972025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
59982025-03-17T16:52:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
59992025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60002025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60012025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60022025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60032025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60042025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
60052025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60062025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60072025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
60082025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
60092025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
60102025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60112025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
60122025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60132025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60142025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
60152025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
60162025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60172025-03-17T16:52:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4413, tv_nsec: 856901695 } })
60182025-03-17T16:52:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4413, tv_nsec: 856902745 } })
60192025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
60202025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.425021ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
60212025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
60222025-03-17T16:52:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.527183ms iteration = 12 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
60232025-03-17T16:52:06.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60242025-03-17T16:52:06.595ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
60252025-03-17T16:52:06.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
60262025-03-17T16:52:06.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
60272025-03-17T16:52:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4413, tv_nsec: 859507161 } })
60282025-03-17T16:52:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60292025-03-17T16:52:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
60302025-03-17T16:52:06.597ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:42:06.593378871 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
60312025-03-17T16:52:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.567307ms iteration = 12 status = {"expiration":"2025-03-17T16:42:06.593378871Z","pruned":[]}
60322025-03-17T16:52:06.598ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.767512ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
60332025-03-17T16:52:06.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
60342025-03-17T16:52:06.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60352025-03-17T16:52:06.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60362025-03-17T16:52:06.601ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
60372025-03-17T16:52:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60382025-03-17T16:52:06.603ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60392025-03-17T16:52:06.603ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
60402025-03-17T16:52:06.603ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60412025-03-17T16:52:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60422025-03-17T16:52:06.603ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60432025-03-17T16:52:06.604ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60442025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
60452025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60462025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
60472025-03-17T16:52:06.606ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4413, tv_nsec: 868991134 } })
60482025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60492025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60502025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60512025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
60522025-03-17T16:52:06.606ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
60532025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60542025-03-17T16:52:06.606ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 13.348276ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
60552025-03-17T16:52:06.608ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60562025-03-17T16:52:06.608ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60572025-03-17T16:52:06.608ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60582025-03-17T16:52:06.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60592025-03-17T16:52:06.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
60602025-03-17T16:52:06.609ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60612025-03-17T16:52:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60622025-03-17T16:52:06.611ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
60632025-03-17T16:52:06.611ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
60642025-03-17T16:52:06.611ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 18.037666ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
60652025-03-17T16:52:06.612ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60662025-03-17T16:52:06.612ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60672025-03-17T16:52:06.612ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60682025-03-17T16:52:06.612ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60692025-03-17T16:52:06.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60702025-03-17T16:52:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
60712025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
60722025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60732025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
60742025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
60752025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
60762025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
60772025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60782025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
60792025-03-17T16:52:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
60802025-03-17T16:52:06.618ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60812025-03-17T16:52:06.619ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60822025-03-17T16:52:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60832025-03-17T16:52:06.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
60842025-03-17T16:52:06.625ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 31.405002ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
60852025-03-17T16:52:06.625ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60862025-03-17T16:52:06.625ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
60872025-03-17T16:52:06.625ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60882025-03-17T16:52:06.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60892025-03-17T16:52:06.627ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60902025-03-17T16:52:06.630ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60912025-03-17T16:52:06.630ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60922025-03-17T16:52:06.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60932025-03-17T16:52:06.633ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4413, tv_nsec: 895800748 } })
60942025-03-17T16:52:06.633ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
60952025-03-17T16:52:06.637ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60962025-03-17T16:52:06.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60972025-03-17T16:52:06.638ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60982025-03-17T16:52:06.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60992025-03-17T16:52:06.651ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 58.280038ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
61002025-03-17T16:52:06.653ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61012025-03-17T16:52:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
61022025-03-17T16:52:06.656ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61032025-03-17T16:52:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61042025-03-17T16:52:06.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61052025-03-17T16:52:06.656ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61062025-03-17T16:52:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
61072025-03-17T16:52:06.659ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
61082025-03-17T16:52:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
61092025-03-17T16:52:06.661ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
61102025-03-17T16:52:06.661ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61112025-03-17T16:52:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61122025-03-17T16:52:06.663ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
61132025-03-17T16:52:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
61142025-03-17T16:52:06.667ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 73.972775ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
61152025-03-17T16:52:06.668ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61162025-03-17T16:52:06.669ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61172025-03-17T16:52:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61182025-03-17T16:52:06.671ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
61192025-03-17T16:52:06.671ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
61202025-03-17T16:52:06.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
61212025-03-17T16:52:06.672ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61222025-03-17T16:52:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61232025-03-17T16:52:06.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61242025-03-17T16:52:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61252025-03-17T16:52:06.679ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
61262025-03-17T16:52:06.679ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
61272025-03-17T16:52:06.679ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
61282025-03-17T16:52:06.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61292025-03-17T16:52:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61302025-03-17T16:52:06.681ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61312025-03-17T16:52:06.683ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 90.291384ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61322025-03-17T16:52:06.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61332025-03-17T16:52:06.686ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61342025-03-17T16:52:06.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61352025-03-17T16:52:06.687ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
61362025-03-17T16:52:06.687ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
61372025-03-17T16:52:06.687ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
61382025-03-17T16:52:06.687ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61392025-03-17T16:52:06.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
61402025-03-17T16:52:06.689ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61412025-03-17T16:52:06.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61422025-03-17T16:52:06.693ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61432025-03-17T16:52:06.693ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61442025-03-17T16:52:06.694ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
61452025-03-17T16:52:06.694ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
61462025-03-17T16:52:06.694ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61472025-03-17T16:52:06.694ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.382889ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
61482025-03-17T16:52:06.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61492025-03-17T16:52:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
61502025-03-17T16:52:06.697ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61512025-03-17T16:52:06.697ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61522025-03-17T16:52:06.697ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
61532025-03-17T16:52:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61542025-03-17T16:52:06.697ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
61552025-03-17T16:52:06.697ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
61562025-03-17T16:52:06.697ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 104.308284ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
61572025-03-17T16:52:06.697ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 81.540787ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
61582025-03-17T16:52:06.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
61592025-03-17T16:52:06.698ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
61602025-03-17T16:52:06.698ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61612025-03-17T16:52:06.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61622025-03-17T16:52:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61632025-03-17T16:52:06.701ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61642025-03-17T16:52:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61652025-03-17T16:52:06.703ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61662025-03-17T16:52:06.706ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
61672025-03-17T16:52:06.706ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
61682025-03-17T16:52:06.706ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
61692025-03-17T16:52:06.706ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 113.352698ms iteration = 12 status = {"generation":2}
61702025-03-17T16:52:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
61712025-03-17T16:52:06.707ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61722025-03-17T16:52:06.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
61732025-03-17T16:52:06.713ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
61742025-03-17T16:52:06.714ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 120.682274ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
61752025-03-17T16:52:06.728ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
61762025-03-17T16:52:06.728ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 134.877819ms iteration = 25 status = {"physical_disks_added":0}
61772025-03-17T16:52:06.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
61782025-03-17T16:52:06.729ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61792025-03-17T16:52:06.730ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
61802025-03-17T16:52:06.730ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
61812025-03-17T16:52:06.730ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 136.503323ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
61822025-03-17T16:52:06.730ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
61832025-03-17T16:52:06.730ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 136.748199ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
61842025-03-17T16:52:06.731ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61852025-03-17T16:52:06.733ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
61862025-03-17T16:52:06.733ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61872025-03-17T16:52:06.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61882025-03-17T16:52:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
61892025-03-17T16:52:06.735ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61902025-03-17T16:52:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61912025-03-17T16:52:06.737ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 144.092656ms iteration = 23 status = 0
61922025-03-17T16:52:06.740ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
61932025-03-17T16:52:06.740ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 146.519278ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
61942025-03-17T16:52:06.747ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
61952025-03-17T16:52:06.747ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 154.160561ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
61962025-03-17T16:52:06.754ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
61972025-03-17T16:52:06.754ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
61982025-03-17T16:52:06.754ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61992025-03-17T16:52:06.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62002025-03-17T16:52:06.767ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 174.303753ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
62012025-03-17T16:52:06.768ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
62022025-03-17T16:52:06.769ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62032025-03-17T16:52:06.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62042025-03-17T16:52:06.771ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62052025-03-17T16:52:06.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62062025-03-17T16:52:06.814ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
62072025-03-17T16:52:06.814ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62082025-03-17T16:52:06.815ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "133e3903-69df-4a92-b5f5-aff313cdd2e2", "content-length": "26", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62092025-03-17T16:52:06.815ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
62102025-03-17T16:52:06.817ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
62112025-03-17T16:52:06.932ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
62122025-03-17T16:52:06.933ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "06d99068-0c61-45d3-9575-85ceefffe5ba", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
62132025-03-17T16:52:06.933ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
62142025-03-17T16:52:06.934ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "19777f07-f8d3-4050-b959-a8422baa3c54", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
62152025-03-17T16:52:06.934ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
62162025-03-17T16:52:06.936ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
62172025-03-17T16:52:06.936ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 343.154749ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
62182025-03-17T16:52:06.943ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
62192025-03-17T16:52:06.943ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
62202025-03-17T16:52:06.943ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "55805bc5-9c8e-46e7-ad3f-cc6657740b05", "content-length": "372", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62212025-03-17T16:52:06.943ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62222025-03-17T16:52:06.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62232025-03-17T16:52:06.944ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62242025-03-17T16:52:06.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
62252025-03-17T16:52:06.948ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
62262025-03-17T16:52:06.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
62272025-03-17T16:52:06.951ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
62282025-03-17T16:52:06.951ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
62292025-03-17T16:52:06.951ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
62302025-03-17T16:52:06.951ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 358.318034ms iteration = 12 status = {"generation":1}
62312025-03-17T16:52:06.974ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62322025-03-17T16:52:06.974ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
62332025-03-17T16:52:06.976ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
62342025-03-17T16:52:06.976ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
62352025-03-17T16:52:06.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
62362025-03-17T16:52:06.995ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
62372025-03-17T16:52:06.995ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62382025-03-17T16:52:06.995ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4be3caaa-4530-461d-86f7-125ee69dbca7", "content-length": "26", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62392025-03-17T16:52:06.995ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
62402025-03-17T16:52:06.997ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
62412025-03-17T16:52:07.001ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
62422025-03-17T16:52:07.001ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
62432025-03-17T16:52:07.001ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ee9d5ed3-8c1a-48c6-8ca5-9dc51cd9b40d", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62442025-03-17T16:52:07.001ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62452025-03-17T16:52:07.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62462025-03-17T16:52:07.003ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
62472025-03-17T16:52:07.003ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62482025-03-17T16:52:07.003ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "996746d1-a24c-47e7-a38f-b85fd49f3c4e", "content-length": "26", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62492025-03-17T16:52:07.003ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
62502025-03-17T16:52:07.004ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62512025-03-17T16:52:07.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62522025-03-17T16:52:07.005ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
62532025-03-17T16:52:07.007ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62542025-03-17T16:52:07.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62552025-03-17T16:52:07.011ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
62562025-03-17T16:52:07.011ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
62572025-03-17T16:52:07.011ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a8e83e03-7cc2-48dc-8fd0-323515e680f4", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62582025-03-17T16:52:07.012ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
62592025-03-17T16:52:07.012ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 419.29488ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
62602025-03-17T16:52:07.012ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 419.24909ms iteration = 23 status = {}
62612025-03-17T16:52:07.045ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62622025-03-17T16:52:07.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62632025-03-17T16:52:07.047ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62642025-03-17T16:52:07.049ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
62652025-03-17T16:52:07.049ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62662025-03-17T16:52:07.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62672025-03-17T16:52:07.050ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62682025-03-17T16:52:07.052ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
62692025-03-17T16:52:07.052ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62702025-03-17T16:52:07.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
62712025-03-17T16:52:07.105ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
62722025-03-17T16:52:07.106ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
62732025-03-17T16:52:07.106ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f0f5750-6835-4d2f-9aaf-ec64584378e0", "content-length": "1", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62742025-03-17T16:52:07.107ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62752025-03-17T16:52:07.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
62762025-03-17T16:52:07.110ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 516.739558ms iteration = 23 status = null
62772025-03-17T16:52:07.114ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 521.123801ms iteration = 23 status = {}
62782025-03-17T16:52:07.181ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62792025-03-17T16:52:07.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62802025-03-17T16:52:07.182ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
62812025-03-17T16:52:07.182ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
62822025-03-17T16:52:07.182ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "282ae9a0-0dc2-43e6-b22e-ed8b56c55b0d", "date": "Mon, 17 Mar 2025 16:52:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62832025-03-17T16:52:07.182ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62842025-03-17T16:52:07.183ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8522699-bfa6-4b14-97e2-e3e8c8aa3f15", "content-length": "691", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
62852025-03-17T16:52:07.183ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
62862025-03-17T16:52:07.183ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
62872025-03-17T16:52:07.183ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
62882025-03-17T16:52:07.184ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 590.894256ms iteration = 23 status = {}
62892025-03-17T16:52:07.184ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "9ef548db-bcb4-42d6-ade7-92931ad997f5", "date": "Mon, 17 Mar 2025 16:52:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62902025-03-17T16:52:07.184ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62912025-03-17T16:52:07.184ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
62922025-03-17T16:52:07.184ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62932025-03-17T16:52:07.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62942025-03-17T16:52:07.186ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "600df94b-764e-4d13-a97e-3e25c991d2f0", "date": "Mon, 17 Mar 2025 16:52:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62952025-03-17T16:52:07.186ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62962025-03-17T16:52:07.188ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62972025-03-17T16:52:07.188ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 594.742528ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
62982025-03-17T16:52:07.196ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
62992025-03-17T16:52:07.197ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c6c8edb-dc9a-4a3b-b06f-b7ec86721bcd", "content-length": "134", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
63002025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
63012025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
63022025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
63032025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
63042025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63052025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63062025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63072025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
63082025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
63092025-03-17T16:52:07.197ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
63102025-03-17T16:52:07.198ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
63112025-03-17T16:52:07.198ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63122025-03-17T16:52:07.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63132025-03-17T16:52:07.200ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
63142025-03-17T16:52:07.201ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "364b35dd-2491-4fe0-bbc8-defaaf388b80", "content-length": "2", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
63152025-03-17T16:52:07.201ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
63162025-03-17T16:52:07.201ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "0577193d-ec40-4e9c-9c6c-4c268d9d79d2", "content-length": "52", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
63172025-03-17T16:52:07.201ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
63182025-03-17T16:52:07.201ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
63192025-03-17T16:52:07.302ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "abbc0ef2-406c-4f8a-861d-1240e6d95b4d", "date": "Mon, 17 Mar 2025 16:52:06 GMT"} })
63202025-03-17T16:52:07.302ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
63212025-03-17T16:52:07.302ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
63222025-03-17T16:52:07.302ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c917442-c125-4a0f-b161-8fa7564626f9", "content-length": "682", "date": "Mon, 17 Mar 2025 16:52:07 GMT"} })
63232025-03-17T16:52:07.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63242025-03-17T16:52:07.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63252025-03-17T16:52:07.307ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63262025-03-17T16:52:07.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63272025-03-17T16:52:07.312ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63282025-03-17T16:52:07.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63292025-03-17T16:52:07.314ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63302025-03-17T16:52:07.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63312025-03-17T16:52:07.322ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 728.321399ms iteration = 23 status = {}
63322025-03-17T16:52:07.643ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
63332025-03-17T16:52:07.643ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
63342025-03-17T16:52:07.700ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
63352025-03-17T16:52:07.700ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:52076
63362025-03-17T16:52:07.700ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:52076 req_id = d8d8eafc-681b-41b4-91c0-4f3ed1776ea6 roles = RoleSet { roles: {} } uri = /metrics/producers
63372025-03-17T16:52:07.701ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:52076 req_id = d8d8eafc-681b-41b4-91c0-4f3ed1776ea6 resource = Database result = Ok(()) uri = /metrics/producers
63382025-03-17T16:52:07.751ZINFOnexus: assigned collector to new producer collector_id = 1374f180-03c9-4b69-abf3-a0c92428bb9a file = nexus/src/app/oximeter.rs:76 producer_id = 807f84c2-ce3d-4190-8b1c-85a3b322601f
63392025-03-17T16:52:07.751ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 50499 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:52076 req_id = d8d8eafc-681b-41b4-91c0-4f3ed1776ea6 response_code = 201 uri = /metrics/producers
63402025-03-17T16:52:07.751ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "d8d8eafc-681b-41b4-91c0-4f3ed1776ea6", "content-length": "41", "date": "Mon, 17 Mar 2025 16:52:07 GMT"} })
63412025-03-17T16:52:07.752ZDEBGnexus (producer-registration-task): registered with nexus successfully
63422025-03-17T16:52:07.753ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
63432025-03-17T16:52:10.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 263 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = dfbb952f-4cea-403f-9278-4b9fd969e8f8 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
63442025-03-17T16:52:16.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
63452025-03-17T16:52:16.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63462025-03-17T16:52:16.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63472025-03-17T16:52:16.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
63482025-03-17T16:52:16.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63492025-03-17T16:52:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63502025-03-17T16:52:16.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63512025-03-17T16:52:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63522025-03-17T16:52:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63532025-03-17T16:52:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63542025-03-17T16:52:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63552025-03-17T16:52:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63562025-03-17T16:52:16.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63572025-03-17T16:52:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
63582025-03-17T16:52:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63592025-03-17T16:52:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63602025-03-17T16:52:16.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
63612025-03-17T16:52:16.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.43749ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
63622025-03-17T16:52:16.629ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63632025-03-17T16:52:16.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63642025-03-17T16:52:16.631ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.604925ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
63652025-03-17T16:52:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = ba1f4cf4-523f-4fc0-a226-6e4ed5eaac1e response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
63662025-03-17T16:52:26.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
63672025-03-17T16:52:26.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63682025-03-17T16:52:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63692025-03-17T16:52:26.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
63702025-03-17T16:52:26.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63712025-03-17T16:52:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63722025-03-17T16:52:26.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63732025-03-17T16:52:26.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63742025-03-17T16:52:26.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63752025-03-17T16:52:26.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63762025-03-17T16:52:26.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63772025-03-17T16:52:26.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63782025-03-17T16:52:26.630ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63792025-03-17T16:52:26.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63802025-03-17T16:52:26.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63812025-03-17T16:52:26.635ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
63822025-03-17T16:52:26.635ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63832025-03-17T16:52:26.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63842025-03-17T16:52:26.644ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
63852025-03-17T16:52:26.644ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.305659ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
63862025-03-17T16:52:26.654ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.541207ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
63872025-03-17T16:52:30.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 351 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 07d4c83e-c2e9-4e62-8db3-04b1a99ee246 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
63882025-03-17T16:52:36.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
63892025-03-17T16:52:36.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
63902025-03-17T16:52:36.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
63912025-03-17T16:52:36.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63922025-03-17T16:52:36.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
63932025-03-17T16:52:36.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
63942025-03-17T16:52:36.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
63952025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63962025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
63972025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
63982025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
63992025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
64002025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
64012025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64022025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
64032025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64042025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64052025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
64062025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64072025-03-17T16:52:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
64082025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
64092025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
64102025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64112025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
64122025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
64132025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
64142025-03-17T16:52:36.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
64152025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
64162025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64172025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64182025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
64192025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
64202025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64212025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
64222025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
64232025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
64242025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
64252025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
64262025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64272025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64282025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64292025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64302025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64312025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
64322025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
64332025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64342025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
64352025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
64362025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64372025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64382025-03-17T16:52:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
64392025-03-17T16:52:36.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64402025-03-17T16:52:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4443, tv_nsec: 859435109 } })
64412025-03-17T16:52:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
64422025-03-17T16:52:36.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.757922ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
64432025-03-17T16:52:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4443, tv_nsec: 860405950 } })
64442025-03-17T16:52:36.597ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64452025-03-17T16:52:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
64462025-03-17T16:52:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
64472025-03-17T16:52:36.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64482025-03-17T16:52:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64492025-03-17T16:52:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64502025-03-17T16:52:36.600ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64512025-03-17T16:52:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64522025-03-17T16:52:36.601ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64532025-03-17T16:52:36.602ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64542025-03-17T16:52:36.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64552025-03-17T16:52:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
64562025-03-17T16:52:36.604ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
64572025-03-17T16:52:36.604ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
64582025-03-17T16:52:36.604ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64592025-03-17T16:52:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
64602025-03-17T16:52:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64612025-03-17T16:52:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
64622025-03-17T16:52:36.606ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64632025-03-17T16:52:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64642025-03-17T16:52:36.607ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64652025-03-17T16:52:36.609ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64662025-03-17T16:52:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64672025-03-17T16:52:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
64682025-03-17T16:52:36.609ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64692025-03-17T16:52:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64702025-03-17T16:52:36.609ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64712025-03-17T16:52:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64722025-03-17T16:52:36.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
64732025-03-17T16:52:36.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
64742025-03-17T16:52:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
64752025-03-17T16:52:36.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
64762025-03-17T16:52:36.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
64772025-03-17T16:52:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
64782025-03-17T16:52:36.617ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64792025-03-17T16:52:36.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64802025-03-17T16:52:36.630ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64812025-03-17T16:52:36.630ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
64822025-03-17T16:52:36.630ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64832025-03-17T16:52:36.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64842025-03-17T16:52:36.636ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 42.746645ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
64852025-03-17T16:52:36.637ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64862025-03-17T16:52:36.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64872025-03-17T16:52:36.639ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 45.602106ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64882025-03-17T16:52:36.642ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
64892025-03-17T16:52:36.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
64902025-03-17T16:52:36.644ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 50.467021ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64912025-03-17T16:52:36.645ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
64922025-03-17T16:52:36.645ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64932025-03-17T16:52:36.645ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b369ee1-61ff-4d49-82eb-094bb9707089", "content-length": "26", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
64942025-03-17T16:52:36.645ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64952025-03-17T16:52:36.645ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
64962025-03-17T16:52:36.645ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64972025-03-17T16:52:36.646ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4203e84-29a2-4ecc-b460-9a8877ae4753", "content-length": "26", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
64982025-03-17T16:52:36.646ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64992025-03-17T16:52:36.647ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
65002025-03-17T16:52:36.647ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65012025-03-17T16:52:36.647ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
65022025-03-17T16:52:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65032025-03-17T16:52:36.649ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65042025-03-17T16:52:36.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65052025-03-17T16:52:36.650ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65062025-03-17T16:52:36.651ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65072025-03-17T16:52:36.652ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
65082025-03-17T16:52:36.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
65092025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65102025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
65112025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
65122025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
65132025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65142025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
65152025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65162025-03-17T16:52:36.654ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65172025-03-17T16:52:36.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
65182025-03-17T16:52:36.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
65192025-03-17T16:52:36.657ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
65202025-03-17T16:52:36.657ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
65212025-03-17T16:52:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
65222025-03-17T16:52:36.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65232025-03-17T16:52:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
65242025-03-17T16:52:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65252025-03-17T16:52:36.658ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65262025-03-17T16:52:36.658ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65272025-03-17T16:52:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65282025-03-17T16:52:36.659ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 65.903061ms iteration = 24 status = 0
65292025-03-17T16:52:36.660ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 44.426101ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
65302025-03-17T16:52:36.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
65312025-03-17T16:52:36.660ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65322025-03-17T16:52:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65332025-03-17T16:52:36.665ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 49.157913ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
65342025-03-17T16:52:36.665ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4443, tv_nsec: 928593091 } })
65352025-03-17T16:52:36.666ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
65362025-03-17T16:52:36.666ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65372025-03-17T16:52:36.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
65382025-03-17T16:52:36.669ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
65392025-03-17T16:52:36.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
65402025-03-17T16:52:36.678ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
65412025-03-17T16:52:36.678ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 85.597313ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
65422025-03-17T16:52:36.681ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65432025-03-17T16:52:36.681ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
65442025-03-17T16:52:36.681ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
65452025-03-17T16:52:36.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
65462025-03-17T16:52:36.694ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
65472025-03-17T16:52:36.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
65482025-03-17T16:52:36.696ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 102.604638ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
65492025-03-17T16:52:36.698ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 103.957656ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
65502025-03-17T16:52:36.702ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
65512025-03-17T16:52:36.702ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 108.469643ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
65522025-03-17T16:52:36.716ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
65532025-03-17T16:52:36.716ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
65542025-03-17T16:52:36.716ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 122.688128ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
65552025-03-17T16:52:36.718ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65562025-03-17T16:52:36.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65572025-03-17T16:52:36.727ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65582025-03-17T16:52:36.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65592025-03-17T16:52:36.731ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
65602025-03-17T16:52:36.732ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "89855f74-0790-4a89-a6d3-3da6cd567977", "content-length": "1", "date": "Mon, 17 Mar 2025 16:52:35 GMT"} })
65612025-03-17T16:52:36.732ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 139.60542ms iteration = 24 status = {}
65622025-03-17T16:52:36.733ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
65632025-03-17T16:52:36.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
65642025-03-17T16:52:36.772ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 178.663916ms iteration = 26 status = {"physical_disks_added":0}
65652025-03-17T16:52:36.773ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/e9918c11-2b6e-4498-a828-973f27e877a2/state
65662025-03-17T16:52:36.774ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/e9918c11-2b6e-4498-a828-973f27e877a2/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a5ba2a1-3f62-48d8-9cdf-0f0d58ca99b7", "content-length": "130", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
65672025-03-17T16:52:36.775ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
65682025-03-17T16:52:36.775ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
65692025-03-17T16:52:36.775ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "66ad7a1d-fb61-47f1-97e5-8e878d5331f5", "content-length": "26", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
65702025-03-17T16:52:36.775ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
65712025-03-17T16:52:36.776ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 183.817337ms iteration = 24 status = null
65722025-03-17T16:52:36.776ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
65732025-03-17T16:52:36.779ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = e9e0da6c-8075-4ec2-854f-474653828e7d sled_id = 8007234b-9b03-4ff0-af97-c65524f74b0a state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-17T16:52:29.336421165Z }, migration_in: None, migration_out: None } vmm_id = e9918c11-2b6e-4498-a828-973f27e877a2
65742025-03-17T16:52:36.779ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2088 instance_id = e9e0da6c-8075-4ec2-854f-474653828e7d migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = e9918c11-2b6e-4498-a828-973f27e877a2 sled_id = 8007234b-9b03-4ff0-af97-c65524f74b0a vmm_id = e9918c11-2b6e-4498-a828-973f27e877a2 vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-17T16:52:29.336421165Z }
65752025-03-17T16:52:36.779ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = e9e0da6c-8075-4ec2-854f-474653828e7d roles = RoleSet { roles: {} } sled_id = 8007234b-9b03-4ff0-af97-c65524f74b0a vmm_id = e9918c11-2b6e-4498-a828-973f27e877a2
65762025-03-17T16:52:36.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = e9e0da6c-8075-4ec2-854f-474653828e7d resource = Database result = Ok(()) sled_id = 8007234b-9b03-4ff0-af97-c65524f74b0a vmm_id = e9918c11-2b6e-4498-a828-973f27e877a2
65772025-03-17T16:52:36.798ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
65782025-03-17T16:52:36.798ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65792025-03-17T16:52:36.799ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "623d6654-0287-4de1-90cc-6466165ee91b", "content-length": "585", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
65802025-03-17T16:52:36.799ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65812025-03-17T16:52:36.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65822025-03-17T16:52:36.809ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65832025-03-17T16:52:36.811ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65842025-03-17T16:52:36.811ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65852025-03-17T16:52:36.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65862025-03-17T16:52:36.812ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65872025-03-17T16:52:36.814ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65882025-03-17T16:52:36.814ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65892025-03-17T16:52:36.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65902025-03-17T16:52:36.845ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {2d6f5f4a-5121-423b-be95-ab7b57582a59: {10.151.2.176: {4dc8eed6-0853-4417-9038-2bbcc733155e}, 10.151.2.175: {4dc8eed6-0853-4417-9038-2bbcc733155e}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
65912025-03-17T16:52:36.845ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
65922025-03-17T16:52:36.846ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f82ac09e-d8f0-43be-be55-f38a60f8e3a4", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
65932025-03-17T16:52:36.846ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65942025-03-17T16:52:36.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65952025-03-17T16:52:36.848ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65962025-03-17T16:52:36.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65972025-03-17T16:52:36.851ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65982025-03-17T16:52:36.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65992025-03-17T16:52:36.853ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
66002025-03-17T16:52:36.854ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66012025-03-17T16:52:36.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66022025-03-17T16:52:36.855ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 261.667775ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"running":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
66032025-03-17T16:52:36.856ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66042025-03-17T16:52:36.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66052025-03-17T16:52:36.860ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
66062025-03-17T16:52:36.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
66072025-03-17T16:52:36.864ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
66082025-03-17T16:52:36.864ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
66092025-03-17T16:52:36.864ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
66102025-03-17T16:52:36.864ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2b7ab4d1-ca47-4c78-89d1-f5f6037c0b34", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
66112025-03-17T16:52:36.866ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 272.397774ms iteration = 24 status = {}
66122025-03-17T16:52:36.873ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66132025-03-17T16:52:36.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66142025-03-17T16:52:36.876ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
66152025-03-17T16:52:36.876ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66162025-03-17T16:52:36.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66172025-03-17T16:52:36.887ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
66182025-03-17T16:52:36.887ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "242979ac-5a5b-40fd-a269-960338f0a628", "content-length": "134", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
66192025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
66202025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
66212025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
66222025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
66232025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66242025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66252025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66262025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66272025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
66282025-03-17T16:52:36.887ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
66292025-03-17T16:52:36.888ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
66302025-03-17T16:52:36.888ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66312025-03-17T16:52:36.888ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66322025-03-17T16:52:36.910ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
66332025-03-17T16:52:36.910ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "637c9e20-07ef-4168-ade2-a011bbdb31e1", "content-length": "811", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
66342025-03-17T16:52:36.910ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(11218874) }} sled = sock
66352025-03-17T16:52:36.910ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
66362025-03-17T16:52:36.910ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
66372025-03-17T16:52:36.912ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 318.168504ms iteration = 24 status = {}
66382025-03-17T16:52:36.912ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
66392025-03-17T16:52:36.912ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fbe2606-15e4-48e4-bccb-f7bdbb4f2e85", "content-length": "2", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
66402025-03-17T16:52:36.912ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
66412025-03-17T16:52:36.912ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "f23c47b5-0a4d-4acd-a47e-f7d5466973b6", "content-length": "52", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
66422025-03-17T16:52:36.912ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
66432025-03-17T16:52:36.912ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
66442025-03-17T16:52:37.003ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "37ff11c7-2c23-4f00-abe5-e7e5a3381f75", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
66452025-03-17T16:52:37.003ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
66462025-03-17T16:52:37.003ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
66472025-03-17T16:52:37.004ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "497c9ac6-5625-4b67-be5f-00abef86dc17", "content-length": "682", "date": "Mon, 17 Mar 2025 16:52:36 GMT"} })
66482025-03-17T16:52:37.004ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66492025-03-17T16:52:37.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66502025-03-17T16:52:37.008ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66512025-03-17T16:52:37.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66522025-03-17T16:52:37.011ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66532025-03-17T16:52:37.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66542025-03-17T16:52:37.014ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
66552025-03-17T16:52:37.014ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
66562025-03-17T16:52:37.019ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 425.83812ms iteration = 24 status = {}
66572025-03-17T16:52:40.146ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 11581 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 5635cf92-53b4-4519-91ea-43cb1874a104 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
66582025-03-17T16:52:46.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
66592025-03-17T16:52:46.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66602025-03-17T16:52:46.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66612025-03-17T16:52:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
66622025-03-17T16:52:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66632025-03-17T16:52:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66642025-03-17T16:52:46.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66652025-03-17T16:52:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66662025-03-17T16:52:46.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66672025-03-17T16:52:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66682025-03-17T16:52:46.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66692025-03-17T16:52:46.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66702025-03-17T16:52:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66712025-03-17T16:52:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
66722025-03-17T16:52:46.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66732025-03-17T16:52:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66742025-03-17T16:52:46.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
66752025-03-17T16:52:46.614ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.698934ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
66762025-03-17T16:52:46.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66772025-03-17T16:52:46.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66782025-03-17T16:52:46.653ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.046226ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66792025-03-17T16:52:50.125ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:47125
66802025-03-17T16:52:50.125ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47125 req_id = b57de412-a074-4112-9a63-f62a8bfb743c roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
66812025-03-17T16:52:50.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47125 req_id = b57de412-a074-4112-9a63-f62a8bfb743c resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
66822025-03-17T16:52:50.128ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3274 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47125 req_id = b57de412-a074-4112-9a63-f62a8bfb743c response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
66832025-03-17T16:52:50.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47125 req_id = ea2d2b87-6f7f-443d-9556-ab75af554c2c roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
66842025-03-17T16:52:50.129ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47125 req_id = ea2d2b87-6f7f-443d-9556-ab75af554c2c resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
66852025-03-17T16:52:50.131ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2663 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:47125 req_id = ea2d2b87-6f7f-443d-9556-ab75af554c2c response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
66862025-03-17T16:52:50.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 509 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = fd591a6e-fba9-4d0b-9dd9-69bfeae0d9b7 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
66872025-03-17T16:52:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
66882025-03-17T16:52:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66892025-03-17T16:52:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66902025-03-17T16:52:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
66912025-03-17T16:52:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66922025-03-17T16:52:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66932025-03-17T16:52:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66942025-03-17T16:52:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66952025-03-17T16:52:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66962025-03-17T16:52:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66972025-03-17T16:52:56.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66982025-03-17T16:52:56.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66992025-03-17T16:52:56.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67002025-03-17T16:52:56.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
67012025-03-17T16:52:56.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67022025-03-17T16:52:56.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67032025-03-17T16:52:56.616ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
67042025-03-17T16:52:56.616ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.413731ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
67052025-03-17T16:52:56.625ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67062025-03-17T16:52:56.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67072025-03-17T16:52:56.655ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.488226ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
67082025-03-17T16:53:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 518 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 6228c616-6ad0-44c6-aefc-f7a63d2fe514 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
67092025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
67102025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
67112025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
67122025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
67132025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
67142025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
67152025-03-17T16:53:06.592ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:43:06.592492712 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
67162025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67172025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
67182025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
67192025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
67202025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67212025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
67222025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
67232025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
67242025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
67252025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
67262025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
67272025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
67282025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
67292025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
67302025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
67312025-03-17T16:53:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
67322025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
67332025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
67342025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67352025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
67362025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67372025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
67382025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
67392025-03-17T16:53:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
67402025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
67412025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
67422025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
67432025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
67442025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67452025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
67462025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67472025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
67482025-03-17T16:53:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4473, tv_nsec: 856017205 } })
67492025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67502025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
67512025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
67522025-03-17T16:53:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
67532025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67542025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67552025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
67562025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
67572025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
67582025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
67592025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67602025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
67612025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
67622025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
67632025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67642025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
67652025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
67662025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
67672025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
67682025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
67692025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
67702025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67712025-03-17T16:53:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
67722025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.611\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
67732025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
67742025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
67752025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67762025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67772025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
67782025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67792025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
67802025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67812025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
67822025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67832025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
67842025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67852025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67862025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67872025-03-17T16:53:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4473, tv_nsec: 856376433 } })
67882025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67892025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
67902025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.425011ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
67912025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
67922025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67932025-03-17T16:53:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67942025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
67952025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67962025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
67972025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
67982025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
67992025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68002025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68012025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
68022025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68032025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68042025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
68052025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68062025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68072025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
68082025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.769018ms iteration = 13 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
68092025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68102025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68112025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68122025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68132025-03-17T16:53:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68142025-03-17T16:53:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4473, tv_nsec: 858858296 } })
68152025-03-17T16:53:06.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
68162025-03-17T16:53:06.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
68172025-03-17T16:53:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4473, tv_nsec: 859347217 } })
68182025-03-17T16:53:06.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68192025-03-17T16:53:06.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
68202025-03-17T16:53:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.604129ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
68212025-03-17T16:53:06.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68222025-03-17T16:53:06.597ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:43:06.592492712 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
68232025-03-17T16:53:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.921246ms iteration = 13 status = {"expiration":"2025-03-17T16:43:06.592492712Z","pruned":[]}
68242025-03-17T16:53:06.598ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68252025-03-17T16:53:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
68262025-03-17T16:53:06.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68272025-03-17T16:53:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68282025-03-17T16:53:06.600ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
68292025-03-17T16:53:06.600ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68302025-03-17T16:53:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68312025-03-17T16:53:06.602ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68322025-03-17T16:53:06.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68332025-03-17T16:53:06.604ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.834242ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
68342025-03-17T16:53:06.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
68352025-03-17T16:53:06.605ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68362025-03-17T16:53:06.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68372025-03-17T16:53:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
68382025-03-17T16:53:06.606ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68392025-03-17T16:53:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68402025-03-17T16:53:06.609ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68412025-03-17T16:53:06.612ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
68422025-03-17T16:53:06.612ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68432025-03-17T16:53:06.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68442025-03-17T16:53:06.616ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
68452025-03-17T16:53:06.616ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
68462025-03-17T16:53:06.616ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
68472025-03-17T16:53:06.616ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
68482025-03-17T16:53:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
68492025-03-17T16:53:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
68502025-03-17T16:53:06.620ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68512025-03-17T16:53:06.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
68522025-03-17T16:53:06.623ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68532025-03-17T16:53:06.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68542025-03-17T16:53:06.625ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68552025-03-17T16:53:06.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
68562025-03-17T16:53:06.633ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68572025-03-17T16:53:06.638ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
68582025-03-17T16:53:06.640ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68592025-03-17T16:53:06.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68602025-03-17T16:53:06.647ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68612025-03-17T16:53:06.647ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
68622025-03-17T16:53:06.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68632025-03-17T16:53:06.649ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
68642025-03-17T16:53:06.649ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
68652025-03-17T16:53:06.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
68662025-03-17T16:53:06.652ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68672025-03-17T16:53:06.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68682025-03-17T16:53:06.654ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68692025-03-17T16:53:06.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68702025-03-17T16:53:06.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68712025-03-17T16:53:06.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68722025-03-17T16:53:06.660ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68732025-03-17T16:53:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68742025-03-17T16:53:06.660ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
68752025-03-17T16:53:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
68762025-03-17T16:53:06.660ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68772025-03-17T16:53:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68782025-03-17T16:53:06.663ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68792025-03-17T16:53:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68802025-03-17T16:53:06.666ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68812025-03-17T16:53:06.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68822025-03-17T16:53:06.668ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
68832025-03-17T16:53:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68842025-03-17T16:53:06.669ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68852025-03-17T16:53:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
68862025-03-17T16:53:06.670ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
68872025-03-17T16:53:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68882025-03-17T16:53:06.670ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 77.301446ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
68892025-03-17T16:53:06.671ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68902025-03-17T16:53:06.671ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
68912025-03-17T16:53:06.671ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68922025-03-17T16:53:06.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68932025-03-17T16:53:06.672ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68942025-03-17T16:53:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
68952025-03-17T16:53:06.672ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
68962025-03-17T16:53:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
68972025-03-17T16:53:06.673ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68982025-03-17T16:53:06.673ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
68992025-03-17T16:53:06.673ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
69002025-03-17T16:53:06.673ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 80.474694ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
69012025-03-17T16:53:06.675ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69022025-03-17T16:53:06.675ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
69032025-03-17T16:53:06.675ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
69042025-03-17T16:53:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
69052025-03-17T16:53:06.675ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69062025-03-17T16:53:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
69072025-03-17T16:53:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69082025-03-17T16:53:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
69092025-03-17T16:53:06.676ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
69102025-03-17T16:53:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
69112025-03-17T16:53:06.676ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
69122025-03-17T16:53:06.676ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
69132025-03-17T16:53:06.676ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69142025-03-17T16:53:06.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
69152025-03-17T16:53:06.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69162025-03-17T16:53:06.677ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
69172025-03-17T16:53:06.677ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
69182025-03-17T16:53:06.677ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
69192025-03-17T16:53:06.677ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69202025-03-17T16:53:06.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69212025-03-17T16:53:06.679ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
69222025-03-17T16:53:06.679ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
69232025-03-17T16:53:06.679ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
69242025-03-17T16:53:06.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
69252025-03-17T16:53:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
69262025-03-17T16:53:06.680ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69272025-03-17T16:53:06.682ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4473, tv_nsec: 944974621 } })
69282025-03-17T16:53:06.682ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
69292025-03-17T16:53:06.682ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69302025-03-17T16:53:06.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69312025-03-17T16:53:06.682ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
69322025-03-17T16:53:06.686ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69332025-03-17T16:53:06.686ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 93.448631ms iteration = 6 status = {"nsuccess":5}
69342025-03-17T16:53:06.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69352025-03-17T16:53:06.689ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69362025-03-17T16:53:06.691ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
69372025-03-17T16:53:06.691ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 99.365478ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
69382025-03-17T16:53:06.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
69392025-03-17T16:53:06.691ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
69402025-03-17T16:53:06.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
69412025-03-17T16:53:06.693ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69422025-03-17T16:53:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
69432025-03-17T16:53:06.697ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69442025-03-17T16:53:06.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69452025-03-17T16:53:06.701ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 85.012651ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
69462025-03-17T16:53:06.701ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 85.231976ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
69472025-03-17T16:53:06.703ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69482025-03-17T16:53:06.703ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 109.738851ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69492025-03-17T16:53:06.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69502025-03-17T16:53:06.705ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69512025-03-17T16:53:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69522025-03-17T16:53:06.710ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
69532025-03-17T16:53:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
69542025-03-17T16:53:06.716ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
69552025-03-17T16:53:06.716ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
69562025-03-17T16:53:06.716ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 123.040595ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
69572025-03-17T16:53:06.719ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
69582025-03-17T16:53:06.719ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69592025-03-17T16:53:06.719ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69602025-03-17T16:53:06.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69612025-03-17T16:53:06.720ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
69622025-03-17T16:53:06.721ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69632025-03-17T16:53:06.721ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 128.223326ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
69642025-03-17T16:53:06.723ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 129.626736ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69652025-03-17T16:53:06.723ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
69662025-03-17T16:53:06.723ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69672025-03-17T16:53:06.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69682025-03-17T16:53:06.723ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
69692025-03-17T16:53:06.723ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69702025-03-17T16:53:06.723ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
69712025-03-17T16:53:06.723ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69722025-03-17T16:53:06.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69732025-03-17T16:53:06.724ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5346ec3-0c07-4a79-890e-104f9f0a679f", "content-length": "26", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
69742025-03-17T16:53:06.724ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69752025-03-17T16:53:06.726ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69762025-03-17T16:53:06.726ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69772025-03-17T16:53:06.728ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
69782025-03-17T16:53:06.728ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 134.774576ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
69792025-03-17T16:53:06.729ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69802025-03-17T16:53:06.729ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
69812025-03-17T16:53:06.729ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
69822025-03-17T16:53:06.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
69832025-03-17T16:53:06.730ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
69842025-03-17T16:53:06.730ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
69852025-03-17T16:53:06.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
69862025-03-17T16:53:06.730ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
69872025-03-17T16:53:06.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
69882025-03-17T16:53:06.732ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
69892025-03-17T16:53:06.732ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69902025-03-17T16:53:06.732ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb458c4-690b-4cac-9923-fa775e2e9e63", "content-length": "26", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
69912025-03-17T16:53:06.732ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69922025-03-17T16:53:06.734ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69932025-03-17T16:53:06.735ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
69942025-03-17T16:53:06.735ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
69952025-03-17T16:53:06.735ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
69962025-03-17T16:53:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
69972025-03-17T16:53:06.737ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69982025-03-17T16:53:06.739ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
69992025-03-17T16:53:06.739ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
70002025-03-17T16:53:06.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
70012025-03-17T16:53:06.743ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
70022025-03-17T16:53:06.743ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
70032025-03-17T16:53:06.743ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
70042025-03-17T16:53:06.743ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 150.691457ms iteration = 13 status = {"generation":1}
70052025-03-17T16:53:06.744ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70062025-03-17T16:53:06.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
70072025-03-17T16:53:06.749ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
70082025-03-17T16:53:06.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
70092025-03-17T16:53:06.756ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 163.362048ms iteration = 25 status = 0
70102025-03-17T16:53:06.759ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70112025-03-17T16:53:06.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70122025-03-17T16:53:06.763ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
70132025-03-17T16:53:06.763ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
70142025-03-17T16:53:06.763ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70152025-03-17T16:53:06.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70162025-03-17T16:53:06.765ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70172025-03-17T16:53:06.765ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70182025-03-17T16:53:06.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70192025-03-17T16:53:06.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
70202025-03-17T16:53:06.771ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
70212025-03-17T16:53:06.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
70222025-03-17T16:53:06.773ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
70232025-03-17T16:53:06.773ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
70242025-03-17T16:53:06.773ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
70252025-03-17T16:53:06.773ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 181.035417ms iteration = 13 status = {"generation":2}
70262025-03-17T16:53:06.775ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
70272025-03-17T16:53:06.775ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 182.263863ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
70282025-03-17T16:53:06.776ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 183.240684ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
70292025-03-17T16:53:06.780ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
70302025-03-17T16:53:06.780ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
70312025-03-17T16:53:06.780ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 187.090277ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
70322025-03-17T16:53:06.799ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70332025-03-17T16:53:06.801ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
70342025-03-17T16:53:06.801ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
70352025-03-17T16:53:06.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
70362025-03-17T16:53:06.812ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70372025-03-17T16:53:06.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70382025-03-17T16:53:06.821ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 227.590834ms iteration = 27 status = {"physical_disks_added":0}
70392025-03-17T16:53:06.824ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
70402025-03-17T16:53:06.824ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 230.939566ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
70412025-03-17T16:53:06.834ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
70422025-03-17T16:53:06.834ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
70432025-03-17T16:53:06.834ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8300f309-1295-4d10-9cf4-46b048b94bfb", "content-length": "26", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
70442025-03-17T16:53:06.834ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
70452025-03-17T16:53:06.836ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
70462025-03-17T16:53:06.893ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
70472025-03-17T16:53:06.916ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 324.03926ms iteration = 25 status = {}
70482025-03-17T16:53:06.949ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
70492025-03-17T16:53:06.949ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
70502025-03-17T16:53:06.949ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a7f7a24-1a05-44ef-9bbc-77bd45273c54", "content-length": "372", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
70512025-03-17T16:53:06.949ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70522025-03-17T16:53:06.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70532025-03-17T16:53:06.951ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70542025-03-17T16:53:06.953ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70552025-03-17T16:53:06.953ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70562025-03-17T16:53:06.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70572025-03-17T16:53:06.955ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70582025-03-17T16:53:06.957ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70592025-03-17T16:53:06.957ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70602025-03-17T16:53:06.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70612025-03-17T16:53:06.965ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
70622025-03-17T16:53:06.965ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
70632025-03-17T16:53:06.965ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b271b2c5-016d-4e88-a335-dd8756b649c3", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
70642025-03-17T16:53:06.965ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70652025-03-17T16:53:06.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70662025-03-17T16:53:06.967ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70672025-03-17T16:53:06.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70682025-03-17T16:53:06.970ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70692025-03-17T16:53:06.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70702025-03-17T16:53:06.974ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
70712025-03-17T16:53:06.974ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
70722025-03-17T16:53:06.974ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "68f752d4-4333-4a69-9ca4-7b8e6b99b019", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
70732025-03-17T16:53:06.976ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 383.087265ms iteration = 25 status = {}
70742025-03-17T16:53:06.982ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
70752025-03-17T16:53:06.983ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c2f9093-eb47-4874-8fcd-6552cbab4910", "content-length": "2", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
70762025-03-17T16:53:06.984ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70772025-03-17T16:53:06.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70782025-03-17T16:53:06.986ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 393.904496ms iteration = 25 status = null
70792025-03-17T16:53:06.989ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
70802025-03-17T16:53:06.989ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 396.677406ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
70812025-03-17T16:53:06.993ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
70822025-03-17T16:53:06.993ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "a46d1128-556e-4c1f-a9eb-b29b3ba8ebfb", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70832025-03-17T16:53:06.993ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70842025-03-17T16:53:06.995ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "7e8a1541-cd0b-4eec-a962-c728040a1a6a", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70852025-03-17T16:53:06.995ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70862025-03-17T16:53:06.996ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70872025-03-17T16:53:06.996ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 404.531474ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
70882025-03-17T16:53:07.010ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
70892025-03-17T16:53:07.011ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "d59be6fc-696c-4999-9cb5-af6da40bcdbe", "date": "Mon, 17 Mar 2025 16:53:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70902025-03-17T16:53:07.011ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70912025-03-17T16:53:07.013ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "498d3720-44b3-4b9a-9b92-dec7e5572c46", "date": "Mon, 17 Mar 2025 16:53:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70922025-03-17T16:53:07.013ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70932025-03-17T16:53:07.015ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "86cae027-88c0-4138-9dea-3744f05c8468", "date": "Mon, 17 Mar 2025 16:53:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70942025-03-17T16:53:07.015ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70952025-03-17T16:53:07.016ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70962025-03-17T16:53:07.016ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 424.347529ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
70972025-03-17T16:53:07.073ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
70982025-03-17T16:53:07.073ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6be6f5ea-f575-43f9-9aa0-3e529081c3f9", "content-length": "691", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
70992025-03-17T16:53:07.073ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
71002025-03-17T16:53:07.073ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
71012025-03-17T16:53:07.073ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
71022025-03-17T16:53:07.075ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 482.428223ms iteration = 25 status = {}
71032025-03-17T16:53:07.100ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71042025-03-17T16:53:07.101ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71052025-03-17T16:53:07.103ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
71062025-03-17T16:53:07.103ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71072025-03-17T16:53:07.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71082025-03-17T16:53:07.113ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
71092025-03-17T16:53:07.113ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "a185f2e9-a7d9-49b0-b26c-d1c867d8d7c5", "content-length": "134", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
71102025-03-17T16:53:07.113ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
71112025-03-17T16:53:07.113ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
71122025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
71132025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
71142025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71152025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71162025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71172025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
71182025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
71192025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
71202025-03-17T16:53:07.114ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
71212025-03-17T16:53:07.114ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71222025-03-17T16:53:07.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71232025-03-17T16:53:07.116ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
71242025-03-17T16:53:07.116ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a9f50a9-603b-41a0-a4d4-702bd828b3ae", "content-length": "2", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
71252025-03-17T16:53:07.116ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
71262025-03-17T16:53:07.116ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "75bc5288-e947-47ab-b392-b3ad045ef4a6", "content-length": "52", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
71272025-03-17T16:53:07.116ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
71282025-03-17T16:53:07.116ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
71292025-03-17T16:53:07.213ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "36fedf6f-bc5d-479a-9351-251514cac7d1", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
71302025-03-17T16:53:07.213ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
71312025-03-17T16:53:07.213ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
71322025-03-17T16:53:07.213ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5cd8e4f-8eff-44ab-b484-6a7b9368c8d2", "content-length": "682", "date": "Mon, 17 Mar 2025 16:53:06 GMT"} })
71332025-03-17T16:53:07.214ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71342025-03-17T16:53:07.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71352025-03-17T16:53:07.217ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71362025-03-17T16:53:07.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71372025-03-17T16:53:07.221ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71382025-03-17T16:53:07.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71392025-03-17T16:53:07.223ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71402025-03-17T16:53:07.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71412025-03-17T16:53:07.230ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 636.8629ms iteration = 25 status = {}
71422025-03-17T16:53:10.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 266 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = ab822464-4850-4673-98d9-220c1a62dcff response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
71432025-03-17T16:53:16.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
71442025-03-17T16:53:16.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71452025-03-17T16:53:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71462025-03-17T16:53:16.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
71472025-03-17T16:53:16.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71482025-03-17T16:53:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71492025-03-17T16:53:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71502025-03-17T16:53:16.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71512025-03-17T16:53:16.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71522025-03-17T16:53:16.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.477086ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
71532025-03-17T16:53:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71542025-03-17T16:53:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71552025-03-17T16:53:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71562025-03-17T16:53:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71572025-03-17T16:53:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71582025-03-17T16:53:16.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71592025-03-17T16:53:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
71602025-03-17T16:53:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71612025-03-17T16:53:16.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71622025-03-17T16:53:16.612ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
71632025-03-17T16:53:16.612ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.912366ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
71642025-03-17T16:53:20.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 396 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 4300ef45-224a-48f9-8500-cfeb50027dae response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
71652025-03-17T16:53:26.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
71662025-03-17T16:53:26.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71672025-03-17T16:53:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71682025-03-17T16:53:26.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
71692025-03-17T16:53:26.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71702025-03-17T16:53:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71712025-03-17T16:53:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71722025-03-17T16:53:26.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71732025-03-17T16:53:26.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71742025-03-17T16:53:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71752025-03-17T16:53:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71762025-03-17T16:53:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71772025-03-17T16:53:26.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71782025-03-17T16:53:26.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71792025-03-17T16:53:26.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71802025-03-17T16:53:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
71812025-03-17T16:53:26.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71822025-03-17T16:53:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71832025-03-17T16:53:26.613ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
71842025-03-17T16:53:26.613ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.674003ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
71852025-03-17T16:53:26.624ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.222769ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
71862025-03-17T16:53:30.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 327 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = a639955c-431e-4b1d-a115-1aa6348a943e response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
71872025-03-17T16:53:36.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
71882025-03-17T16:53:36.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
71892025-03-17T16:53:36.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
71902025-03-17T16:53:36.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
71912025-03-17T16:53:36.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71922025-03-17T16:53:36.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
71932025-03-17T16:53:36.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
71942025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
71952025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
71962025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71972025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
71982025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
71992025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
72002025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
72012025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72022025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72032025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
72042025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
72052025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
72062025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
72072025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
72082025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
72092025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
72102025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
72112025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72122025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
72132025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72142025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
72152025-03-17T16:53:36.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
72162025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72172025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
72182025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
72192025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72202025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72212025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
72222025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
72232025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
72242025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72252025-03-17T16:53:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72262025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72272025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
72282025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72292025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
72302025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72312025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
72322025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72332025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
72342025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72352025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
72362025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72372025-03-17T16:53:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72382025-03-17T16:53:36.595ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4503, tv_nsec: 858342981 } })
72392025-03-17T16:53:36.595ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
72402025-03-17T16:53:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4503, tv_nsec: 859068957 } })
72412025-03-17T16:53:36.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72422025-03-17T16:53:36.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
72432025-03-17T16:53:36.596ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.17627ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
72442025-03-17T16:53:36.597ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72452025-03-17T16:53:36.598ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72462025-03-17T16:53:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
72472025-03-17T16:53:36.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72482025-03-17T16:53:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72492025-03-17T16:53:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72502025-03-17T16:53:36.600ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72512025-03-17T16:53:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72522025-03-17T16:53:36.602ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
72532025-03-17T16:53:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
72542025-03-17T16:53:36.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72552025-03-17T16:53:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
72562025-03-17T16:53:36.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72572025-03-17T16:53:36.603ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72582025-03-17T16:53:36.604ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72592025-03-17T16:53:36.604ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72602025-03-17T16:53:36.606ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
72612025-03-17T16:53:36.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72622025-03-17T16:53:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72632025-03-17T16:53:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
72642025-03-17T16:53:36.607ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72652025-03-17T16:53:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
72662025-03-17T16:53:36.607ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
72672025-03-17T16:53:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72682025-03-17T16:53:36.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
72692025-03-17T16:53:36.609ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72702025-03-17T16:53:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72712025-03-17T16:53:36.610ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72722025-03-17T16:53:36.610ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72732025-03-17T16:53:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72742025-03-17T16:53:36.611ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72752025-03-17T16:53:36.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72762025-03-17T16:53:36.613ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
72772025-03-17T16:53:36.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72782025-03-17T16:53:36.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72792025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72802025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
72812025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
72822025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
72832025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
72842025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
72852025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
72862025-03-17T16:53:36.615ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72872025-03-17T16:53:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72882025-03-17T16:53:36.618ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72892025-03-17T16:53:36.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72902025-03-17T16:53:36.620ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
72912025-03-17T16:53:36.620ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72922025-03-17T16:53:36.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72932025-03-17T16:53:36.632ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
72942025-03-17T16:53:36.632ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 39.924225ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
72952025-03-17T16:53:36.633ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72962025-03-17T16:53:36.633ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72972025-03-17T16:53:36.636ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
72982025-03-17T16:53:36.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72992025-03-17T16:53:36.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
73002025-03-17T16:53:36.636ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 43.176215ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
73012025-03-17T16:53:36.637ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73022025-03-17T16:53:36.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73032025-03-17T16:53:36.638ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
73042025-03-17T16:53:36.638ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73052025-03-17T16:53:36.638ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
73062025-03-17T16:53:36.638ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73072025-03-17T16:53:36.638ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c445287-eed6-424f-9d40-5dd3fac77a55", "content-length": "26", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
73082025-03-17T16:53:36.638ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73092025-03-17T16:53:36.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73102025-03-17T16:53:36.638ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
73112025-03-17T16:53:36.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
73122025-03-17T16:53:36.640ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
73132025-03-17T16:53:36.640ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73142025-03-17T16:53:36.640ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 46.505396ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
73152025-03-17T16:53:36.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73162025-03-17T16:53:36.642ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4503, tv_nsec: 904859147 } })
73172025-03-17T16:53:36.642ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73182025-03-17T16:53:36.642ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
73192025-03-17T16:53:36.643ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 49.762135ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73202025-03-17T16:53:36.645ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
73212025-03-17T16:53:36.645ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73222025-03-17T16:53:36.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73232025-03-17T16:53:36.646ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73242025-03-17T16:53:36.646ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
73252025-03-17T16:53:36.646ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
73262025-03-17T16:53:36.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
73272025-03-17T16:53:36.647ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 54.139829ms iteration = 26 status = 0
73282025-03-17T16:53:36.650ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73292025-03-17T16:53:36.650ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
73302025-03-17T16:53:36.650ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
73312025-03-17T16:53:36.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
73322025-03-17T16:53:36.653ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73332025-03-17T16:53:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73342025-03-17T16:53:36.656ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
73352025-03-17T16:53:36.656ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73362025-03-17T16:53:36.656ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73372025-03-17T16:53:36.656ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "acfdcccb-92ed-4b35-8ce5-cd3344f2c7a2", "content-length": "26", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
73382025-03-17T16:53:36.656ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73392025-03-17T16:53:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
73402025-03-17T16:53:36.658ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73412025-03-17T16:53:36.658ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
73422025-03-17T16:53:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73432025-03-17T16:53:36.660ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 44.842391ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
73442025-03-17T16:53:36.660ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
73452025-03-17T16:53:36.660ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 66.607006ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
73462025-03-17T16:53:36.664ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
73472025-03-17T16:53:36.664ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 69.861776ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
73482025-03-17T16:53:36.665ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 50.196305ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
73492025-03-17T16:53:36.667ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
73502025-03-17T16:53:36.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
73512025-03-17T16:53:36.678ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 85.205725ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73522025-03-17T16:53:36.691ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73532025-03-17T16:53:36.691ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
73542025-03-17T16:53:36.691ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 97.622141ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
73552025-03-17T16:53:36.701ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 107.36466ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
73562025-03-17T16:53:36.720ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73572025-03-17T16:53:36.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73582025-03-17T16:53:36.729ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73592025-03-17T16:53:36.729ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 135.783358ms iteration = 28 status = {"physical_disks_added":0}
73602025-03-17T16:53:36.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73612025-03-17T16:53:36.731ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73622025-03-17T16:53:36.733ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73632025-03-17T16:53:36.733ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73642025-03-17T16:53:36.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73652025-03-17T16:53:36.734ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73662025-03-17T16:53:36.736ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73672025-03-17T16:53:36.736ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73682025-03-17T16:53:36.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73692025-03-17T16:53:36.753ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 160.811114ms iteration = 26 status = {}
73702025-03-17T16:53:36.799ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
73712025-03-17T16:53:36.799ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73722025-03-17T16:53:36.799ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "93fa8bd8-d622-4811-9acd-f73d0fdfc441", "content-length": "372", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
73732025-03-17T16:53:36.799ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73742025-03-17T16:53:36.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73752025-03-17T16:53:36.815ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
73762025-03-17T16:53:36.815ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
73772025-03-17T16:53:36.815ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "829d76af-99e2-42ff-81fd-520a201bad44", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
73782025-03-17T16:53:36.815ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73792025-03-17T16:53:36.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73802025-03-17T16:53:36.817ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73812025-03-17T16:53:36.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73822025-03-17T16:53:36.822ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
73832025-03-17T16:53:36.822ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73842025-03-17T16:53:36.822ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bf29032-a23f-4cb4-aa17-5e645131c72e", "content-length": "26", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
73852025-03-17T16:53:36.822ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73862025-03-17T16:53:36.824ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
73872025-03-17T16:53:36.835ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
73882025-03-17T16:53:36.835ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "7343a88a-8e59-4706-8b74-8bac82b0a26a", "content-length": "2", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
73892025-03-17T16:53:36.836ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
73902025-03-17T16:53:36.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
73912025-03-17T16:53:36.865ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
73922025-03-17T16:53:36.865ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d453f009-42ee-48fa-98f8-04b05925c9e2", "content-length": "691", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
73932025-03-17T16:53:36.865ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
73942025-03-17T16:53:36.865ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
73952025-03-17T16:53:36.865ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
73962025-03-17T16:53:36.866ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 272.412304ms iteration = 26 status = {}
73972025-03-17T16:53:36.867ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 275.098962ms iteration = 26 status = null
73982025-03-17T16:53:36.868ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73992025-03-17T16:53:36.868ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74002025-03-17T16:53:36.871ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
74012025-03-17T16:53:36.871ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
74022025-03-17T16:53:36.871ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d3bdf52f-71b7-4156-a2ed-4edad2a8216e", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
74032025-03-17T16:53:36.872ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 279.149269ms iteration = 26 status = {}
74042025-03-17T16:53:36.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74052025-03-17T16:53:36.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74062025-03-17T16:53:36.941ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
74072025-03-17T16:53:36.941ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74082025-03-17T16:53:36.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74092025-03-17T16:53:36.951ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
74102025-03-17T16:53:36.951ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b2e780d-bd6b-408e-8bd9-60092ea2fab2", "content-length": "134", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
74112025-03-17T16:53:36.951ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
74122025-03-17T16:53:36.951ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
74132025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
74142025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
74152025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74162025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74172025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74182025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74192025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74202025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74212025-03-17T16:53:36.952ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
74222025-03-17T16:53:36.952ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74232025-03-17T16:53:36.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74242025-03-17T16:53:36.955ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
74252025-03-17T16:53:36.955ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ab8de81-cf1f-472a-8128-8ccca6189cc7", "content-length": "2", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
74262025-03-17T16:53:36.955ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
74272025-03-17T16:53:36.955ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "adfdc207-eed8-47e1-8f27-d995e5548d07", "content-length": "52", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
74282025-03-17T16:53:36.955ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
74292025-03-17T16:53:36.955ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
74302025-03-17T16:53:37.055ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f430a519-6492-4e6e-bae2-bac4b7d701dd", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
74312025-03-17T16:53:37.055ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
74322025-03-17T16:53:37.055ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
74332025-03-17T16:53:37.056ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6f9a6f0-f3d6-4930-9e44-8ff25f55026a", "content-length": "682", "date": "Mon, 17 Mar 2025 16:53:36 GMT"} })
74342025-03-17T16:53:37.056ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74352025-03-17T16:53:37.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74362025-03-17T16:53:37.060ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74372025-03-17T16:53:37.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74382025-03-17T16:53:37.064ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74392025-03-17T16:53:37.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74402025-03-17T16:53:37.066ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74412025-03-17T16:53:37.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74422025-03-17T16:53:37.073ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 479.905448ms iteration = 26 status = {}
74432025-03-17T16:53:40.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 256 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 3b3a1da3-2a2e-4ac5-8580-4340e5c9b7cb response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
74442025-03-17T16:53:46.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
74452025-03-17T16:53:46.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74462025-03-17T16:53:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74472025-03-17T16:53:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
74482025-03-17T16:53:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74492025-03-17T16:53:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74502025-03-17T16:53:46.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74512025-03-17T16:53:46.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74522025-03-17T16:53:46.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74532025-03-17T16:53:46.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.996457ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74542025-03-17T16:53:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74552025-03-17T16:53:46.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74562025-03-17T16:53:46.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74572025-03-17T16:53:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74582025-03-17T16:53:46.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74592025-03-17T16:53:46.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74602025-03-17T16:53:46.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
74612025-03-17T16:53:46.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74622025-03-17T16:53:46.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74632025-03-17T16:53:46.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
74642025-03-17T16:53:46.614ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.855928ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
74652025-03-17T16:53:50.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 295 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 23c5d355-728a-4d64-a63a-c037a75a54d1 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
74662025-03-17T16:53:56.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
74672025-03-17T16:53:56.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74682025-03-17T16:53:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74692025-03-17T16:53:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
74702025-03-17T16:53:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74712025-03-17T16:53:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74722025-03-17T16:53:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74732025-03-17T16:53:56.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74742025-03-17T16:53:56.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74752025-03-17T16:53:56.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74762025-03-17T16:53:56.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74772025-03-17T16:53:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74782025-03-17T16:53:56.598ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.827374ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74792025-03-17T16:53:56.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74802025-03-17T16:53:56.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74812025-03-17T16:53:56.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74822025-03-17T16:53:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
74832025-03-17T16:53:56.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74842025-03-17T16:53:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74852025-03-17T16:53:56.611ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
74862025-03-17T16:53:56.611ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.15176ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
74872025-03-17T16:54:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 471 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 9359bbd7-e63a-4537-92e1-7e0b10ca05c0 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
74882025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
74892025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
74902025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
74912025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
74922025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
74932025-03-17T16:54:06.592ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:44:06.592844023 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
74942025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
74952025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
74962025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
74972025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
74982025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
74992025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
75002025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75012025-03-17T16:54:06.592ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
75022025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
75032025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
75042025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
75052025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
75062025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
75072025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
75082025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
75092025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
75102025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
75112025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
75122025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75132025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
75142025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
75152025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
75162025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
75172025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
75182025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
75192025-03-17T16:54:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
75202025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
75212025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
75222025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.59\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
75232025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
75242025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
75252025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75262025-03-17T16:54:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
75272025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
75282025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
75292025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75302025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
75312025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75322025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75332025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
75342025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
75352025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
75362025-03-17T16:54:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
75372025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
75382025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
75392025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
75402025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
75412025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
75422025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
75432025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
75442025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75452025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
75462025-03-17T16:54:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4533, tv_nsec: 856234072 } })
75472025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
75482025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75492025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
75502025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
75512025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
75522025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
75532025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75542025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
75552025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75562025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
75572025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
75582025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
75592025-03-17T16:54:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4533, tv_nsec: 856475027 } })
75602025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
75612025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
75622025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75632025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75642025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75652025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75662025-03-17T16:54:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
75672025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75682025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
75692025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75702025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
75712025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
75722025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
75732025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75742025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
75752025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
75762025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.164265ms iteration = 14 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
75772025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
75782025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75792025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
75802025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.189325ms iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
75812025-03-17T16:54:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
75822025-03-17T16:54:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4533, tv_nsec: 859153675 } })
75832025-03-17T16:54:06.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
75842025-03-17T16:54:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4533, tv_nsec: 859654295 } })
75852025-03-17T16:54:06.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75862025-03-17T16:54:06.596ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
75872025-03-17T16:54:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
75882025-03-17T16:54:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
75892025-03-17T16:54:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.014377ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
75902025-03-17T16:54:06.597ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75912025-03-17T16:54:06.598ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:44:06.592844023 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
75922025-03-17T16:54:06.598ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.096251ms iteration = 14 status = {"expiration":"2025-03-17T16:44:06.592844023Z","pruned":[]}
75932025-03-17T16:54:06.598ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75942025-03-17T16:54:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
75952025-03-17T16:54:06.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75962025-03-17T16:54:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75972025-03-17T16:54:06.600ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
75982025-03-17T16:54:06.600ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
75992025-03-17T16:54:06.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76002025-03-17T16:54:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
76012025-03-17T16:54:06.601ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
76022025-03-17T16:54:06.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
76032025-03-17T16:54:06.602ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 9.145636ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
76042025-03-17T16:54:06.604ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76052025-03-17T16:54:06.604ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76062025-03-17T16:54:06.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76072025-03-17T16:54:06.605ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76082025-03-17T16:54:06.606ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
76092025-03-17T16:54:06.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76102025-03-17T16:54:06.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76112025-03-17T16:54:06.607ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 13.652472ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
76122025-03-17T16:54:06.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
76132025-03-17T16:54:06.608ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76142025-03-17T16:54:06.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76152025-03-17T16:54:06.608ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76162025-03-17T16:54:06.609ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76172025-03-17T16:54:06.610ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76182025-03-17T16:54:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
76192025-03-17T16:54:06.610ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
76202025-03-17T16:54:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
76212025-03-17T16:54:06.611ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
76222025-03-17T16:54:06.611ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
76232025-03-17T16:54:06.611ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 17.431343ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
76242025-03-17T16:54:06.611ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76252025-03-17T16:54:06.611ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76262025-03-17T16:54:06.612ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76272025-03-17T16:54:06.612ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76282025-03-17T16:54:06.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76292025-03-17T16:54:06.614ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
76302025-03-17T16:54:06.614ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76312025-03-17T16:54:06.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76322025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
76332025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76342025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
76352025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
76362025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
76372025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
76382025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
76392025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76402025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
76412025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
76422025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76432025-03-17T16:54:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76442025-03-17T16:54:06.616ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
76452025-03-17T16:54:06.616ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
76462025-03-17T16:54:06.616ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
76472025-03-17T16:54:06.616ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76482025-03-17T16:54:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76492025-03-17T16:54:06.617ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76502025-03-17T16:54:06.619ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4533, tv_nsec: 881726358 } })
76512025-03-17T16:54:06.619ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76522025-03-17T16:54:06.619ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
76532025-03-17T16:54:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76542025-03-17T16:54:06.619ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
76552025-03-17T16:54:06.619ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
76562025-03-17T16:54:06.619ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76572025-03-17T16:54:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
76582025-03-17T16:54:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76592025-03-17T16:54:06.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76602025-03-17T16:54:06.622ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
76612025-03-17T16:54:06.622ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 28.820918ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
76622025-03-17T16:54:06.622ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 6.918038ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
76632025-03-17T16:54:06.622ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76642025-03-17T16:54:06.623ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76652025-03-17T16:54:06.626ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
76662025-03-17T16:54:06.626ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76672025-03-17T16:54:06.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76682025-03-17T16:54:06.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
76692025-03-17T16:54:06.627ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76702025-03-17T16:54:06.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76712025-03-17T16:54:06.627ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
76722025-03-17T16:54:06.627ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76732025-03-17T16:54:06.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76742025-03-17T16:54:06.629ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76752025-03-17T16:54:06.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
76762025-03-17T16:54:06.632ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
76772025-03-17T16:54:06.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
76782025-03-17T16:54:06.635ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76792025-03-17T16:54:06.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76802025-03-17T16:54:06.641ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76812025-03-17T16:54:06.643ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
76822025-03-17T16:54:06.643ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
76832025-03-17T16:54:06.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
76842025-03-17T16:54:06.644ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
76852025-03-17T16:54:06.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
76862025-03-17T16:54:06.644ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
76872025-03-17T16:54:06.644ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.78838ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
76882025-03-17T16:54:06.648ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
76892025-03-17T16:54:06.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
76902025-03-17T16:54:06.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
76912025-03-17T16:54:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
76922025-03-17T16:54:06.656ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
76932025-03-17T16:54:06.656ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
76942025-03-17T16:54:06.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
76952025-03-17T16:54:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
76962025-03-17T16:54:06.658ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
76972025-03-17T16:54:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
76982025-03-17T16:54:06.663ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.754013ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76992025-03-17T16:54:06.663ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77002025-03-17T16:54:06.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77012025-03-17T16:54:06.669ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
77022025-03-17T16:54:06.669ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
77032025-03-17T16:54:06.669ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
77042025-03-17T16:54:06.669ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77052025-03-17T16:54:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77062025-03-17T16:54:06.673ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
77072025-03-17T16:54:06.673ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
77082025-03-17T16:54:06.674ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77092025-03-17T16:54:06.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77102025-03-17T16:54:06.674ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
77112025-03-17T16:54:06.674ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
77122025-03-17T16:54:06.674ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 80.811201ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
77132025-03-17T16:54:06.674ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 59.661358ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
77142025-03-17T16:54:06.678ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
77152025-03-17T16:54:06.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
77162025-03-17T16:54:06.744ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 150.562644ms iteration = 29 status = {"physical_disks_added":0}
77172025-03-17T16:54:06.758ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
77182025-03-17T16:54:06.765ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
77192025-03-17T16:54:06.765ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
77202025-03-17T16:54:06.765ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "16cee0fa-9f97-4b7d-9040-19a9a667089d", "content-length": "26", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
77212025-03-17T16:54:06.765ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
77222025-03-17T16:54:06.766ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
77232025-03-17T16:54:06.896ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
77242025-03-17T16:54:06.896ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
77252025-03-17T16:54:06.896ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b1fddeb-35d4-49d4-bddf-32ff7d31620b", "content-length": "26", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
77262025-03-17T16:54:06.896ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
77272025-03-17T16:54:06.897ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
77282025-03-17T16:54:06.905ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
77292025-03-17T16:54:06.928ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
77302025-03-17T16:54:06.946ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
77312025-03-17T16:54:06.946ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
77322025-03-17T16:54:06.946ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf0320e7-aa50-4755-8fa1-2208b70d6a04", "content-length": "372", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
77332025-03-17T16:54:06.946ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77342025-03-17T16:54:06.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77352025-03-17T16:54:06.947ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
77362025-03-17T16:54:06.947ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5adaeca-ba7e-486b-8c13-04947e2a0a3f", "content-length": "2", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
77372025-03-17T16:54:06.949ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
77382025-03-17T16:54:06.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
77392025-03-17T16:54:06.949ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 356.449954ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
77402025-03-17T16:54:06.951ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
77412025-03-17T16:54:06.951ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
77422025-03-17T16:54:06.951ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
77432025-03-17T16:54:06.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
77442025-03-17T16:54:06.951ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77452025-03-17T16:54:06.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
77462025-03-17T16:54:06.952ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77472025-03-17T16:54:06.953ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
77482025-03-17T16:54:06.953ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
77492025-03-17T16:54:06.953ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ce9bbb-420f-4191-9ab9-3a45992b1830", "content-length": "26", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
77502025-03-17T16:54:06.953ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 360.627444ms iteration = 27 status = null
77512025-03-17T16:54:06.953ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
77522025-03-17T16:54:06.955ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
77532025-03-17T16:54:06.955ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
77542025-03-17T16:54:06.955ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77552025-03-17T16:54:06.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77562025-03-17T16:54:06.958ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77572025-03-17T16:54:06.958ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
77582025-03-17T16:54:06.958ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
77592025-03-17T16:54:06.958ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
77602025-03-17T16:54:06.958ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 366.026979ms iteration = 14 status = {"generation":2}
77612025-03-17T16:54:06.961ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
77622025-03-17T16:54:06.961ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77632025-03-17T16:54:06.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
77642025-03-17T16:54:06.963ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 369.866441ms iteration = 27 status = 0
77652025-03-17T16:54:06.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77662025-03-17T16:54:06.984ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77672025-03-17T16:54:06.986ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
77682025-03-17T16:54:06.986ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77692025-03-17T16:54:06.986ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
77702025-03-17T16:54:06.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77712025-03-17T16:54:06.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
77722025-03-17T16:54:06.987ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
77732025-03-17T16:54:06.987ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
77742025-03-17T16:54:06.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
77752025-03-17T16:54:06.988ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 395.321336ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
77762025-03-17T16:54:06.990ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
77772025-03-17T16:54:06.990ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
77782025-03-17T16:54:06.990ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 396.736256ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
77792025-03-17T16:54:06.991ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 397.638006ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
77802025-03-17T16:54:06.991ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77812025-03-17T16:54:06.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77822025-03-17T16:54:06.992ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77832025-03-17T16:54:06.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
77842025-03-17T16:54:06.995ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77852025-03-17T16:54:06.995ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
77862025-03-17T16:54:06.995ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
77872025-03-17T16:54:06.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77882025-03-17T16:54:06.995ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d050913c-0f1b-4fbb-8a8d-8c80f7cd6309", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
77892025-03-17T16:54:06.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77902025-03-17T16:54:06.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77912025-03-17T16:54:06.997ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77922025-03-17T16:54:06.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77932025-03-17T16:54:07.001ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77942025-03-17T16:54:07.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77952025-03-17T16:54:07.005ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
77962025-03-17T16:54:07.005ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
77972025-03-17T16:54:07.005ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "73774bbf-f08b-4455-9cac-3874ef0f1b88", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
77982025-03-17T16:54:07.008ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 414.753602ms iteration = 27 status = {}
77992025-03-17T16:54:07.024ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
78002025-03-17T16:54:07.024ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 431.345039ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
78012025-03-17T16:54:07.028ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
78022025-03-17T16:54:07.028ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
78032025-03-17T16:54:07.028ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78042025-03-17T16:54:07.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78052025-03-17T16:54:07.030ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78062025-03-17T16:54:07.031ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
78072025-03-17T16:54:07.031ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 438.054782ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
78082025-03-17T16:54:07.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
78092025-03-17T16:54:07.033ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78102025-03-17T16:54:07.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78112025-03-17T16:54:07.035ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
78122025-03-17T16:54:07.035ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 442.325393ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
78132025-03-17T16:54:07.036ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 443.784475ms iteration = 27 status = {}
78142025-03-17T16:54:07.037ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
78152025-03-17T16:54:07.037ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
78162025-03-17T16:54:07.037ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
78172025-03-17T16:54:07.037ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 444.403308ms iteration = 14 status = {"generation":1}
78182025-03-17T16:54:07.038ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
78192025-03-17T16:54:07.038ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "0d5d62c9-27a5-4c59-b4c1-1341ade9ccfa", "date": "Mon, 17 Mar 2025 16:54:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78202025-03-17T16:54:07.038ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78212025-03-17T16:54:07.040ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "46c16cfa-d3e6-4844-936e-44adf3b20ec3", "date": "Mon, 17 Mar 2025 16:54:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78222025-03-17T16:54:07.040ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78232025-03-17T16:54:07.042ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
78242025-03-17T16:54:07.042ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 449.564019ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
78252025-03-17T16:54:07.045ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
78262025-03-17T16:54:07.045ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "53fb3978-3dc2-4ce3-a897-484437a73fec", "date": "Mon, 17 Mar 2025 16:54:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
78272025-03-17T16:54:07.045ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
78282025-03-17T16:54:07.047ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "e2b71c08-53d3-4db0-be92-b78899e7926a", "date": "Mon, 17 Mar 2025 16:54:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
78292025-03-17T16:54:07.047ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
78302025-03-17T16:54:07.049ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "3c8019f7-735e-4b48-9af2-5d35a637fa3b", "date": "Mon, 17 Mar 2025 16:54:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
78312025-03-17T16:54:07.049ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
78322025-03-17T16:54:07.052ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
78332025-03-17T16:54:07.052ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 459.170385ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
78342025-03-17T16:54:07.059ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78352025-03-17T16:54:07.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78362025-03-17T16:54:07.089ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78372025-03-17T16:54:07.091ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
78382025-03-17T16:54:07.091ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78392025-03-17T16:54:07.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78402025-03-17T16:54:07.093ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78412025-03-17T16:54:07.095ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
78422025-03-17T16:54:07.095ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78432025-03-17T16:54:07.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78442025-03-17T16:54:07.129ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78452025-03-17T16:54:07.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78462025-03-17T16:54:07.131ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
78472025-03-17T16:54:07.131ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78482025-03-17T16:54:07.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78492025-03-17T16:54:07.142ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
78502025-03-17T16:54:07.142ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "73fcb2c6-108a-49e3-b7dc-83d4332a220c", "content-length": "134", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
78512025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
78522025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
78532025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
78542025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
78552025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78562025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78572025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78582025-03-17T16:54:07.142ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
78592025-03-17T16:54:07.143ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
78602025-03-17T16:54:07.143ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
78612025-03-17T16:54:07.143ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
78622025-03-17T16:54:07.143ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78632025-03-17T16:54:07.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78642025-03-17T16:54:07.145ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
78652025-03-17T16:54:07.145ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
78662025-03-17T16:54:07.145ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "439a9f09-4ee6-4c0a-a15f-fa4741f6856e", "content-length": "2", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
78672025-03-17T16:54:07.145ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
78682025-03-17T16:54:07.145ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a37c993-ab15-4127-b802-f0dcaf41b15c", "content-length": "52", "date": "Mon, 17 Mar 2025 16:54:07 GMT"} })
78692025-03-17T16:54:07.145ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
78702025-03-17T16:54:07.145ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
78712025-03-17T16:54:07.145ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "671e7424-79b8-4ff1-883f-5e9aaf2cbce3", "content-length": "691", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
78722025-03-17T16:54:07.145ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
78732025-03-17T16:54:07.145ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
78742025-03-17T16:54:07.145ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
78752025-03-17T16:54:07.146ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 553.037315ms iteration = 27 status = {}
78762025-03-17T16:54:07.236ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1d1d9f56-46c6-4f61-b263-712cafdce3e0", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
78772025-03-17T16:54:07.236ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
78782025-03-17T16:54:07.236ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
78792025-03-17T16:54:07.236ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3ace3b7-c4d7-445a-b558-e4dcf8f5a723", "content-length": "682", "date": "Mon, 17 Mar 2025 16:54:06 GMT"} })
78802025-03-17T16:54:07.236ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78812025-03-17T16:54:07.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78822025-03-17T16:54:07.240ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78832025-03-17T16:54:07.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78842025-03-17T16:54:07.243ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78852025-03-17T16:54:07.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78862025-03-17T16:54:07.245ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78872025-03-17T16:54:07.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78882025-03-17T16:54:07.253ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 659.884163ms iteration = 27 status = {}
78892025-03-17T16:54:10.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 321 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 405f9002-24a6-4fb0-9b78-3610e2e87646 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
78902025-03-17T16:54:16.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
78912025-03-17T16:54:16.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78922025-03-17T16:54:16.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78932025-03-17T16:54:16.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
78942025-03-17T16:54:16.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78952025-03-17T16:54:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78962025-03-17T16:54:16.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78972025-03-17T16:54:16.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
78982025-03-17T16:54:16.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78992025-03-17T16:54:16.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79002025-03-17T16:54:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79012025-03-17T16:54:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79022025-03-17T16:54:16.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79032025-03-17T16:54:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
79042025-03-17T16:54:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79052025-03-17T16:54:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79062025-03-17T16:54:16.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
79072025-03-17T16:54:16.614ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.332978ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
79082025-03-17T16:54:16.629ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79092025-03-17T16:54:16.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79102025-03-17T16:54:16.631ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.219379ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
79112025-03-17T16:54:20.124ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:36485
79122025-03-17T16:54:20.124ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36485 req_id = 51f63d2d-9304-4852-a0ed-fe5922488270 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
79132025-03-17T16:54:20.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36485 req_id = 51f63d2d-9304-4852-a0ed-fe5922488270 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
79142025-03-17T16:54:20.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3378 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36485 req_id = 51f63d2d-9304-4852-a0ed-fe5922488270 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
79152025-03-17T16:54:20.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36485 req_id = 2b6156a2-2612-4e47-b660-477772bd53bc roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
79162025-03-17T16:54:20.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36485 req_id = 2b6156a2-2612-4e47-b660-477772bd53bc resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
79172025-03-17T16:54:20.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2598 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:36485 req_id = 2b6156a2-2612-4e47-b660-477772bd53bc response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
79182025-03-17T16:54:20.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 5955eb3a-7180-48e6-b063-0ce891496ebb response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
79192025-03-17T16:54:26.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
79202025-03-17T16:54:26.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79212025-03-17T16:54:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79222025-03-17T16:54:26.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
79232025-03-17T16:54:26.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79242025-03-17T16:54:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79252025-03-17T16:54:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79262025-03-17T16:54:26.601ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
79272025-03-17T16:54:26.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79282025-03-17T16:54:26.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79292025-03-17T16:54:26.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79302025-03-17T16:54:26.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79312025-03-17T16:54:26.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79322025-03-17T16:54:26.608ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
79332025-03-17T16:54:26.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79342025-03-17T16:54:26.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79352025-03-17T16:54:26.619ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
79362025-03-17T16:54:26.619ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.507958ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
79372025-03-17T16:54:26.628ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79382025-03-17T16:54:26.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79392025-03-17T16:54:26.658ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.414669ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
79402025-03-17T16:54:30.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 386 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 65b30203-73aa-4a55-b56e-d4a362fdb50e response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
79412025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
79422025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
79432025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
79442025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
79452025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
79462025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
79472025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
79482025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
79492025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
79502025-03-17T16:54:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
79512025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
79522025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
79532025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79542025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79552025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
79562025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
79572025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79582025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
79592025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
79602025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
79612025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
79622025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79632025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
79642025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
79652025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
79662025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79672025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
79682025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
79692025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
79702025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
79712025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79722025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
79732025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
79742025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
79752025-03-17T16:54:36.594ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
79762025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79772025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
79782025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
79792025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79802025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
79812025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79822025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79832025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
79842025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79852025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
79862025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
79872025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79882025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79892025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79902025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
79912025-03-17T16:54:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
79922025-03-17T16:54:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4563, tv_nsec: 859070802 } })
79932025-03-17T16:54:36.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
79942025-03-17T16:54:36.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4563, tv_nsec: 859852429 } })
79952025-03-17T16:54:36.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.22559ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
79962025-03-17T16:54:36.597ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79972025-03-17T16:54:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
79982025-03-17T16:54:36.599ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79992025-03-17T16:54:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
80002025-03-17T16:54:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
80012025-03-17T16:54:36.599ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80022025-03-17T16:54:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80032025-03-17T16:54:36.602ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80042025-03-17T16:54:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
80052025-03-17T16:54:36.604ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80062025-03-17T16:54:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80072025-03-17T16:54:36.606ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80082025-03-17T16:54:36.608ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80092025-03-17T16:54:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80102025-03-17T16:54:36.609ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4563, tv_nsec: 871950838 } })
80112025-03-17T16:54:36.609ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
80122025-03-17T16:54:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
80132025-03-17T16:54:36.609ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
80142025-03-17T16:54:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
80152025-03-17T16:54:36.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
80162025-03-17T16:54:36.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
80172025-03-17T16:54:36.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
80182025-03-17T16:54:36.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
80192025-03-17T16:54:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
80202025-03-17T16:54:36.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
80212025-03-17T16:54:36.642ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80222025-03-17T16:54:36.642ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80232025-03-17T16:54:36.643ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
80242025-03-17T16:54:36.643ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
80252025-03-17T16:54:36.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
80262025-03-17T16:54:36.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
80272025-03-17T16:54:36.646ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
80282025-03-17T16:54:36.646ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
80292025-03-17T16:54:36.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
80302025-03-17T16:54:36.646ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
80312025-03-17T16:54:36.646ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80322025-03-17T16:54:36.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80332025-03-17T16:54:36.646ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80342025-03-17T16:54:36.648ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 54.16181ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
80352025-03-17T16:54:36.649ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80362025-03-17T16:54:36.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80372025-03-17T16:54:36.650ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 55.879597ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
80382025-03-17T16:54:36.650ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80392025-03-17T16:54:36.650ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
80402025-03-17T16:54:36.650ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
80412025-03-17T16:54:36.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80422025-03-17T16:54:36.651ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
80432025-03-17T16:54:36.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
80442025-03-17T16:54:36.651ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80452025-03-17T16:54:36.651ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80462025-03-17T16:54:36.651ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
80472025-03-17T16:54:36.651ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80482025-03-17T16:54:36.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80492025-03-17T16:54:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
80502025-03-17T16:54:36.654ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
80512025-03-17T16:54:36.654ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
80522025-03-17T16:54:36.654ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80532025-03-17T16:54:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
80542025-03-17T16:54:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80552025-03-17T16:54:36.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
80562025-03-17T16:54:36.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
80572025-03-17T16:54:36.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80582025-03-17T16:54:36.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80592025-03-17T16:54:36.657ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80602025-03-17T16:54:36.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80612025-03-17T16:54:36.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
80622025-03-17T16:54:36.660ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80632025-03-17T16:54:36.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80642025-03-17T16:54:36.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
80652025-03-17T16:54:36.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80662025-03-17T16:54:36.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80672025-03-17T16:54:36.663ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 47.860525ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
80682025-03-17T16:54:36.665ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
80692025-03-17T16:54:36.665ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 70.760876ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
80702025-03-17T16:54:36.665ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 49.779306ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
80712025-03-17T16:54:36.671ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80722025-03-17T16:54:36.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80732025-03-17T16:54:36.673ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
80742025-03-17T16:54:36.673ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80752025-03-17T16:54:36.674ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "39a30d24-6542-4395-befd-7b9da0d611c1", "content-length": "26", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
80762025-03-17T16:54:36.674ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80772025-03-17T16:54:36.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
80782025-03-17T16:54:36.674ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
80792025-03-17T16:54:36.674ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
80802025-03-17T16:54:36.674ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80812025-03-17T16:54:36.674ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 81.503366ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
80822025-03-17T16:54:36.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
80832025-03-17T16:54:36.674ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ca14e0-cacc-4778-bbb2-a634e199ce22", "content-length": "26", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
80842025-03-17T16:54:36.674ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80852025-03-17T16:54:36.677ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
80862025-03-17T16:54:36.677ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
80872025-03-17T16:54:36.679ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 85.677035ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
80882025-03-17T16:54:36.685ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 91.586862ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
80892025-03-17T16:54:36.688ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 94.230698ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
80902025-03-17T16:54:36.689ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
80912025-03-17T16:54:36.689ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
80922025-03-17T16:54:36.689ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "72efe3a5-4d6f-4f6b-8d80-d9e6884d7a04", "content-length": "26", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
80932025-03-17T16:54:36.689ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80942025-03-17T16:54:36.689ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
80952025-03-17T16:54:36.689ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
80962025-03-17T16:54:36.689ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
80972025-03-17T16:54:36.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
80982025-03-17T16:54:36.691ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
80992025-03-17T16:54:36.692ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81002025-03-17T16:54:36.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
81012025-03-17T16:54:36.694ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81022025-03-17T16:54:36.696ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
81032025-03-17T16:54:36.696ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81042025-03-17T16:54:36.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
81052025-03-17T16:54:36.704ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81062025-03-17T16:54:36.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
81072025-03-17T16:54:36.707ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81082025-03-17T16:54:36.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81092025-03-17T16:54:36.710ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 115.952444ms iteration = 28 status = 0
81102025-03-17T16:54:36.718ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81112025-03-17T16:54:36.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81122025-03-17T16:54:36.729ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
81132025-03-17T16:54:36.729ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
81142025-03-17T16:54:36.729ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 134.9569ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
81152025-03-17T16:54:36.739ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
81162025-03-17T16:54:36.739ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 145.587818ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
81172025-03-17T16:54:36.753ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 159.604138ms iteration = 30 status = {"physical_disks_added":0}
81182025-03-17T16:54:36.772ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 179.879253ms iteration = 28 status = {}
81192025-03-17T16:54:36.773ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
81202025-03-17T16:54:36.774ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "50c82459-d735-4bae-84b7-0c4d72f37455", "content-length": "2", "date": "Mon, 17 Mar 2025 16:54:35 GMT"} })
81212025-03-17T16:54:36.775ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
81222025-03-17T16:54:36.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
81232025-03-17T16:54:36.777ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 184.098802ms iteration = 28 status = null
81242025-03-17T16:54:36.811ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
81252025-03-17T16:54:36.811ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
81262025-03-17T16:54:36.812ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4672c0af-54b4-42c1-ad68-7b9db7ff884c", "content-length": "372", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81272025-03-17T16:54:36.812ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81282025-03-17T16:54:36.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81292025-03-17T16:54:36.814ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81302025-03-17T16:54:36.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81312025-03-17T16:54:36.815ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81322025-03-17T16:54:36.817ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
81332025-03-17T16:54:36.817ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81342025-03-17T16:54:36.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81352025-03-17T16:54:36.819ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81362025-03-17T16:54:36.821ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
81372025-03-17T16:54:36.821ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81382025-03-17T16:54:36.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81392025-03-17T16:54:36.862ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
81402025-03-17T16:54:36.862ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
81412025-03-17T16:54:36.862ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0837a783-1090-45d8-a005-7e08b3694bee", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81422025-03-17T16:54:36.862ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81432025-03-17T16:54:36.862ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81442025-03-17T16:54:36.865ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81452025-03-17T16:54:36.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81462025-03-17T16:54:36.868ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81472025-03-17T16:54:36.868ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81482025-03-17T16:54:36.871ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
81492025-03-17T16:54:36.871ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
81502025-03-17T16:54:36.871ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "8271264a-53d5-4530-bac4-a7812451802e", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81512025-03-17T16:54:36.873ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 278.838872ms iteration = 28 status = {}
81522025-03-17T16:54:36.880ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
81532025-03-17T16:54:36.881ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6039d81b-7e03-4317-b089-9a0fea3d82a5", "content-length": "691", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81542025-03-17T16:54:36.881ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
81552025-03-17T16:54:36.881ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
81562025-03-17T16:54:36.881ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
81572025-03-17T16:54:36.882ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 288.55083ms iteration = 28 status = {}
81582025-03-17T16:54:36.885ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81592025-03-17T16:54:36.885ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81602025-03-17T16:54:36.888ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
81612025-03-17T16:54:36.888ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81622025-03-17T16:54:36.888ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81632025-03-17T16:54:36.898ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
81642025-03-17T16:54:36.898ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "07d6a25e-5514-4359-b198-09cbdd7748a0", "content-length": "134", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81652025-03-17T16:54:36.898ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
81662025-03-17T16:54:36.898ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
81672025-03-17T16:54:36.898ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
81682025-03-17T16:54:36.898ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
81692025-03-17T16:54:36.898ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81702025-03-17T16:54:36.899ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81712025-03-17T16:54:36.899ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81722025-03-17T16:54:36.899ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81732025-03-17T16:54:36.899ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
81742025-03-17T16:54:36.899ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
81752025-03-17T16:54:36.899ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
81762025-03-17T16:54:36.899ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81772025-03-17T16:54:36.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81782025-03-17T16:54:36.901ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
81792025-03-17T16:54:36.901ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "e091bc40-d2e8-4ebf-ae7c-4e6547c1e4f8", "content-length": "2", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81802025-03-17T16:54:36.901ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
81812025-03-17T16:54:36.901ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c63419e-7a06-4ba8-b250-84984f0defc9", "content-length": "52", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81822025-03-17T16:54:36.901ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
81832025-03-17T16:54:36.901ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
81842025-03-17T16:54:36.999ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4f8ddf62-0476-42a6-b2d4-5b7dded44778", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81852025-03-17T16:54:36.999ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
81862025-03-17T16:54:36.999ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
81872025-03-17T16:54:36.999ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "60ae035d-b141-4dc7-89dd-762c59b71ed8", "content-length": "682", "date": "Mon, 17 Mar 2025 16:54:36 GMT"} })
81882025-03-17T16:54:36.999ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81892025-03-17T16:54:36.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81902025-03-17T16:54:37.004ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81912025-03-17T16:54:37.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81922025-03-17T16:54:37.007ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81932025-03-17T16:54:37.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81942025-03-17T16:54:37.010ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81952025-03-17T16:54:37.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81962025-03-17T16:54:37.018ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 423.846567ms iteration = 28 status = {}
81972025-03-17T16:54:37.753ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
81982025-03-17T16:54:37.753ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
81992025-03-17T16:54:37.785ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
82002025-03-17T16:54:37.785ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:39261
82012025-03-17T16:54:37.785ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:39261 req_id = 238508cd-90fe-456a-b33d-a6cbc084b141 roles = RoleSet { roles: {} } uri = /metrics/producers
82022025-03-17T16:54:37.786ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:39261 req_id = 238508cd-90fe-456a-b33d-a6cbc084b141 resource = Database result = Ok(()) uri = /metrics/producers
82032025-03-17T16:54:37.833ZINFOnexus: assigned collector to new producer collector_id = 1374f180-03c9-4b69-abf3-a0c92428bb9a file = nexus/src/app/oximeter.rs:76 producer_id = 807f84c2-ce3d-4190-8b1c-85a3b322601f
82042025-03-17T16:54:37.833ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 47986 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:39261 req_id = 238508cd-90fe-456a-b33d-a6cbc084b141 response_code = 201 uri = /metrics/producers
82052025-03-17T16:54:37.833ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "238508cd-90fe-456a-b33d-a6cbc084b141", "content-length": "41", "date": "Mon, 17 Mar 2025 16:54:37 GMT"} })
82062025-03-17T16:54:37.836ZDEBGnexus (producer-registration-task): registered with nexus successfully
82072025-03-17T16:54:37.836ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
82082025-03-17T16:54:40.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 4b56022f-37c7-4fe6-8d82-9d52bcdb386e response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
82092025-03-17T16:54:46.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 83 reason = Timeout
82102025-03-17T16:54:46.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82112025-03-17T16:54:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82122025-03-17T16:54:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 83 reason = Timeout
82132025-03-17T16:54:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82142025-03-17T16:54:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82152025-03-17T16:54:46.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82162025-03-17T16:54:46.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82172025-03-17T16:54:46.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82182025-03-17T16:54:46.597ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.373224ms iteration = 83 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82192025-03-17T16:54:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82202025-03-17T16:54:46.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82212025-03-17T16:54:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82222025-03-17T16:54:46.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82232025-03-17T16:54:46.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82242025-03-17T16:54:46.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82252025-03-17T16:54:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
82262025-03-17T16:54:46.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82272025-03-17T16:54:46.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82282025-03-17T16:54:46.611ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
82292025-03-17T16:54:46.611ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.66751ms iteration = 83 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
82302025-03-17T16:54:50.135ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 291 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = a17a7d41-b50f-46dc-9783-5d17bf179459 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
82312025-03-17T16:54:56.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 84 reason = Timeout
82322025-03-17T16:54:56.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82332025-03-17T16:54:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82342025-03-17T16:54:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 84 reason = Timeout
82352025-03-17T16:54:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82362025-03-17T16:54:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82372025-03-17T16:54:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82382025-03-17T16:54:56.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82392025-03-17T16:54:56.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82402025-03-17T16:54:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82412025-03-17T16:54:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82422025-03-17T16:54:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82432025-03-17T16:54:56.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82442025-03-17T16:54:56.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82452025-03-17T16:54:56.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82462025-03-17T16:54:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
82472025-03-17T16:54:56.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82482025-03-17T16:54:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82492025-03-17T16:54:56.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
82502025-03-17T16:54:56.614ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.088061ms iteration = 84 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
82512025-03-17T16:54:56.624ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.096626ms iteration = 84 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82522025-03-17T16:55:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 496 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = ede51233-58dd-411a-9a36-09ee517d2b7e response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
82532025-03-17T16:55:05.124ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:62656
82542025-03-17T16:55:05.124ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = b018432b-5a03-4fac-89f6-61d71c03603b roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
82552025-03-17T16:55:05.125ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = b018432b-5a03-4fac-89f6-61d71c03603b resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
82562025-03-17T16:55:05.128ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3148 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = b018432b-5a03-4fac-89f6-61d71c03603b response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
82572025-03-17T16:55:05.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = a9888747-fb15-4997-b718-6f1657d01800 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
82582025-03-17T16:55:05.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = a9888747-fb15-4997-b718-6f1657d01800 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
82592025-03-17T16:55:05.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2115 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = a9888747-fb15-4997-b718-6f1657d01800 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
82602025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 17 reason = Timeout
82612025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 15 reason = Timeout
82622025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
82632025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
82642025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 85 reason = Timeout
82652025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 29 reason = Timeout
82662025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
82672025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 17 reason = Timeout
82682025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82692025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 15 reason = Timeout
82702025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 29 reason = Timeout
82712025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
82722025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 15 reason = Timeout
82732025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 15 reason = Timeout
82742025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 15 reason = Timeout
82752025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 15 reason = Timeout
82762025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
82772025-03-17T16:55:06.592ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:45:06.592942635 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
82782025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
82792025-03-17T16:55:06.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 29 reason = Timeout
82802025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
82812025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
82822025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
82832025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
82842025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82852025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
82862025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 16 reason = Timeout
82872025-03-17T16:55:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
82882025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 15 reason = Timeout
82892025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82902025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
82912025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 29 reason = Timeout
82922025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.74\u{b5}s iteration = 16 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
82932025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 29 reason = Timeout
82942025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
82952025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
82962025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 15 reason = Timeout
82972025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 29 reason = Timeout
82982025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
82992025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
83002025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 29 reason = Timeout
83012025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 31 reason = Timeout
83022025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83032025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83042025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 15 reason = Timeout
83052025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83062025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
83072025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
83082025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 29 reason = Timeout
83092025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 29 reason = Timeout
83102025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
83112025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83122025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 85 reason = Timeout
83132025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83142025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
83152025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 15 reason = Timeout
83162025-03-17T16:55:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
83172025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
83182025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 29 reason = Timeout
83192025-03-17T16:55:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
83202025-03-17T16:55:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4593, tv_nsec: 856337192 } })
83212025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83222025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 29 reason = Timeout
83232025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83242025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 29 reason = Timeout
83252025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
83262025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 29 reason = Timeout
83272025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 29 reason = Timeout
83282025-03-17T16:55:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
83292025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83302025-03-17T16:55:06.594ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4593, tv_nsec: 856438254 } })
83312025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
83322025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83332025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
83342025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
83352025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83362025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
83372025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83382025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83392025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
83402025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
83412025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83422025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83432025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83442025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
83452025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
83462025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
83472025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.096994ms iteration = 15 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
83482025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83492025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83502025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
83512025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.200145ms iteration = 15 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
83522025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
83532025-03-17T16:55:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
83542025-03-17T16:55:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4593, tv_nsec: 859624063 } })
83552025-03-17T16:55:06.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83562025-03-17T16:55:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
83572025-03-17T16:55:06.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
83582025-03-17T16:55:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
83592025-03-17T16:55:06.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4593, tv_nsec: 860911890 } })
83602025-03-17T16:55:06.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
83612025-03-17T16:55:06.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
83622025-03-17T16:55:06.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83632025-03-17T16:55:06.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83642025-03-17T16:55:06.600ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:45:06.592942635 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
83652025-03-17T16:55:06.600ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 7.083982ms iteration = 15 status = {"expiration":"2025-03-17T16:45:06.592942635Z","pruned":[]}
83662025-03-17T16:55:06.600ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.186414ms iteration = 29 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
83672025-03-17T16:55:06.603ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83682025-03-17T16:55:06.604ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83692025-03-17T16:55:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
83702025-03-17T16:55:06.607ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83712025-03-17T16:55:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83722025-03-17T16:55:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83732025-03-17T16:55:06.607ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83742025-03-17T16:55:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83752025-03-17T16:55:06.609ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83762025-03-17T16:55:06.610ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83772025-03-17T16:55:06.610ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
83782025-03-17T16:55:06.610ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83792025-03-17T16:55:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83802025-03-17T16:55:06.610ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83812025-03-17T16:55:06.611ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 18.132099ms iteration = 15 status = {"errors":[],"found_port_ok":[]}
83822025-03-17T16:55:06.612ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
83832025-03-17T16:55:06.612ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
83842025-03-17T16:55:06.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83852025-03-17T16:55:06.613ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83862025-03-17T16:55:06.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83872025-03-17T16:55:06.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83882025-03-17T16:55:06.615ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83892025-03-17T16:55:06.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 29 reason = Timeout
83902025-03-17T16:55:06.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 29 reason = Timeout
83912025-03-17T16:55:06.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
83922025-03-17T16:55:06.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
83932025-03-17T16:55:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
83942025-03-17T16:55:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
83952025-03-17T16:55:06.616ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83962025-03-17T16:55:06.617ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83972025-03-17T16:55:06.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83982025-03-17T16:55:06.618ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4593, tv_nsec: 881113853 } })
83992025-03-17T16:55:06.618ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
84002025-03-17T16:55:06.618ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84012025-03-17T16:55:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84022025-03-17T16:55:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
84032025-03-17T16:55:06.619ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
84042025-03-17T16:55:06.634ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84052025-03-17T16:55:06.637ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
84062025-03-17T16:55:06.637ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84072025-03-17T16:55:06.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84082025-03-17T16:55:06.641ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84092025-03-17T16:55:06.645ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
84102025-03-17T16:55:06.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
84112025-03-17T16:55:06.645ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
84122025-03-17T16:55:06.646ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84132025-03-17T16:55:06.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84142025-03-17T16:55:06.648ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84152025-03-17T16:55:06.648ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
84162025-03-17T16:55:06.648ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
84172025-03-17T16:55:06.648ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 55.374516ms iteration = 15 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
84182025-03-17T16:55:06.649ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84192025-03-17T16:55:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84202025-03-17T16:55:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
84212025-03-17T16:55:06.650ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
84222025-03-17T16:55:06.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
84232025-03-17T16:55:06.651ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84242025-03-17T16:55:06.653ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 59.732789ms iteration = 29 status = {"errors":[],"garbage_collect_requested":[]}
84252025-03-17T16:55:06.654ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
84262025-03-17T16:55:06.654ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
84272025-03-17T16:55:06.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
84282025-03-17T16:55:06.655ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84292025-03-17T16:55:06.656ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84302025-03-17T16:55:06.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
84312025-03-17T16:55:06.657ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
84322025-03-17T16:55:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
84332025-03-17T16:55:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84342025-03-17T16:55:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
84352025-03-17T16:55:06.659ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84362025-03-17T16:55:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84372025-03-17T16:55:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
84382025-03-17T16:55:06.660ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
84392025-03-17T16:55:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
84402025-03-17T16:55:06.662ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 69.087409ms iteration = 31 status = {"physical_disks_added":0}
84412025-03-17T16:55:06.662ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 47.15174ms iteration = 29 status = {"errors":[],"requests_created_ok":[]}
84422025-03-17T16:55:06.663ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
84432025-03-17T16:55:06.663ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 70.212944ms iteration = 85 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
84442025-03-17T16:55:06.663ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.741474ms iteration = 85 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
84452025-03-17T16:55:06.663ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84462025-03-17T16:55:06.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84472025-03-17T16:55:06.673ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84482025-03-17T16:55:06.674ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
84492025-03-17T16:55:06.674ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
84502025-03-17T16:55:06.674ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
84512025-03-17T16:55:06.674ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
84522025-03-17T16:55:06.674ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
84532025-03-17T16:55:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
84542025-03-17T16:55:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
84552025-03-17T16:55:06.676ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
84562025-03-17T16:55:06.676ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
84572025-03-17T16:55:06.677ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
84582025-03-17T16:55:06.677ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
84592025-03-17T16:55:06.677ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84602025-03-17T16:55:06.677ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
84612025-03-17T16:55:06.677ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84622025-03-17T16:55:06.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
84632025-03-17T16:55:06.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84642025-03-17T16:55:06.678ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
84652025-03-17T16:55:06.678ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
84662025-03-17T16:55:06.678ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
84672025-03-17T16:55:06.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84682025-03-17T16:55:06.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
84692025-03-17T16:55:06.679ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
84702025-03-17T16:55:06.679ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 86.320649ms iteration = 29 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
84712025-03-17T16:55:06.680ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.811568ms iteration = 29 status = {"errors":[],"finish_invoked_ok":[]}
84722025-03-17T16:55:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
84732025-03-17T16:55:06.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
84742025-03-17T16:55:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
84752025-03-17T16:55:06.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
84762025-03-17T16:55:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
84772025-03-17T16:55:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
84782025-03-17T16:55:06.694ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
84792025-03-17T16:55:06.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
84802025-03-17T16:55:06.696ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
84812025-03-17T16:55:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
84822025-03-17T16:55:06.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84832025-03-17T16:55:06.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84842025-03-17T16:55:06.699ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
84852025-03-17T16:55:06.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
84862025-03-17T16:55:06.700ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84872025-03-17T16:55:06.700ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84882025-03-17T16:55:06.700ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84892025-03-17T16:55:06.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84902025-03-17T16:55:06.703ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
84912025-03-17T16:55:06.703ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
84922025-03-17T16:55:06.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
84932025-03-17T16:55:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
84942025-03-17T16:55:06.705ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84952025-03-17T16:55:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84962025-03-17T16:55:06.707ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
84972025-03-17T16:55:06.707ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
84982025-03-17T16:55:06.707ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 113.537172ms iteration = 15 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
84992025-03-17T16:55:06.709ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 115.743469ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
85002025-03-17T16:55:06.719ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
85012025-03-17T16:55:06.719ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
85022025-03-17T16:55:06.719ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
85032025-03-17T16:55:06.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
85042025-03-17T16:55:06.721ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85052025-03-17T16:55:06.724ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
85062025-03-17T16:55:06.724ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
85072025-03-17T16:55:06.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
85082025-03-17T16:55:06.726ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 132.648801ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
85092025-03-17T16:55:06.726ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 133.163892ms iteration = 29 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
85102025-03-17T16:55:06.727ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
85112025-03-17T16:55:06.727ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
85122025-03-17T16:55:06.727ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
85132025-03-17T16:55:06.727ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 134.290416ms iteration = 15 status = {"generation":2}
85142025-03-17T16:55:06.730ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
85152025-03-17T16:55:06.730ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
85162025-03-17T16:55:06.730ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 136.683648ms iteration = 29 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
85172025-03-17T16:55:06.733ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
85182025-03-17T16:55:06.734ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
85192025-03-17T16:55:06.734ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85202025-03-17T16:55:06.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85212025-03-17T16:55:06.737ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
85222025-03-17T16:55:06.740ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
85232025-03-17T16:55:06.741ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 147.53199ms iteration = 29 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
85242025-03-17T16:55:06.744ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85252025-03-17T16:55:06.744ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
85262025-03-17T16:55:06.744ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
85272025-03-17T16:55:06.745ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbf0a772-460b-41f4-bd68-77e0d4795cc2", "content-length": "26", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
85282025-03-17T16:55:06.745ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
85292025-03-17T16:55:06.747ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
85302025-03-17T16:55:06.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
85312025-03-17T16:55:06.747ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85322025-03-17T16:55:06.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85332025-03-17T16:55:06.748ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85342025-03-17T16:55:06.749ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
85352025-03-17T16:55:06.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
85362025-03-17T16:55:06.751ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
85372025-03-17T16:55:06.751ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
85382025-03-17T16:55:06.751ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
85392025-03-17T16:55:06.751ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 158.528275ms iteration = 15 status = {"generation":1}
85402025-03-17T16:55:06.751ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85412025-03-17T16:55:06.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
85422025-03-17T16:55:06.751ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
85432025-03-17T16:55:06.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
85442025-03-17T16:55:06.754ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
85452025-03-17T16:55:06.754ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
85462025-03-17T16:55:06.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
85472025-03-17T16:55:06.754ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 161.172302ms iteration = 29 status = 0
85482025-03-17T16:55:06.763ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
85492025-03-17T16:55:06.763ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
85502025-03-17T16:55:06.763ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3307ff0-caef-4f51-b338-bce4e31cfdbb", "content-length": "26", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
85512025-03-17T16:55:06.763ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
85522025-03-17T16:55:06.764ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
85532025-03-17T16:55:06.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85542025-03-17T16:55:06.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85552025-03-17T16:55:06.794ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
85562025-03-17T16:55:06.794ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 200.713759ms iteration = 15 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
85572025-03-17T16:55:06.822ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
85582025-03-17T16:55:06.909ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
85592025-03-17T16:55:06.909ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bfceb39-6547-477e-adaa-328facdabad4", "content-length": "2", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
85602025-03-17T16:55:06.910ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
85612025-03-17T16:55:06.910ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "458bef6f-3841-433e-ae39-04308c85838c", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
85622025-03-17T16:55:06.910ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
85632025-03-17T16:55:06.911ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
85642025-03-17T16:55:06.911ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
85652025-03-17T16:55:06.913ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "be5b609f-2c05-4643-835d-2c989a3077f0", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
85662025-03-17T16:55:06.913ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
85672025-03-17T16:55:06.914ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
85682025-03-17T16:55:06.914ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
85692025-03-17T16:55:06.914ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0562210-a1b9-4432-b7e4-d9d30e418e44", "content-length": "372", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
85702025-03-17T16:55:06.914ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
85712025-03-17T16:55:06.915ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
85722025-03-17T16:55:06.915ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
85732025-03-17T16:55:06.915ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 322.862035ms iteration = 17 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
85742025-03-17T16:55:06.917ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 324.668754ms iteration = 29 status = null
85752025-03-17T16:55:06.926ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
85762025-03-17T16:55:06.926ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
85772025-03-17T16:55:06.926ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e6e822b-9f96-4396-a215-ffaddb68c7e7", "content-length": "26", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
85782025-03-17T16:55:06.926ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
85792025-03-17T16:55:06.929ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 336.135059ms iteration = 29 status = {}
85802025-03-17T16:55:06.929ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
85812025-03-17T16:55:06.954ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
85822025-03-17T16:55:06.954ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 361.280167ms iteration = 15 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
85832025-03-17T16:55:06.961ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
85842025-03-17T16:55:06.961ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
85852025-03-17T16:55:06.962ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c5016a79-e1e5-49ef-94c3-c09001cff242", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
85862025-03-17T16:55:06.962ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
85872025-03-17T16:55:06.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
85882025-03-17T16:55:06.964ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
85892025-03-17T16:55:06.965ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
85902025-03-17T16:55:06.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
85912025-03-17T16:55:06.968ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
85922025-03-17T16:55:06.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
85932025-03-17T16:55:06.971ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
85942025-03-17T16:55:06.971ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
85952025-03-17T16:55:06.971ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "16fc33f4-b681-471c-ab35-8a1903be1e6c", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
85962025-03-17T16:55:06.972ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 379.409136ms iteration = 29 status = {}
85972025-03-17T16:55:07.004ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85982025-03-17T16:55:07.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85992025-03-17T16:55:07.006ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86002025-03-17T16:55:07.008ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
86012025-03-17T16:55:07.008ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86022025-03-17T16:55:07.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86032025-03-17T16:55:07.010ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86042025-03-17T16:55:07.012ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
86052025-03-17T16:55:07.012ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86062025-03-17T16:55:07.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86072025-03-17T16:55:07.028ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
86082025-03-17T16:55:07.029ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "5abd3c0e-3b6e-4563-9fc0-ceb0d9b83d5f", "date": "Mon, 17 Mar 2025 16:55:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
86092025-03-17T16:55:07.029ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
86102025-03-17T16:55:07.030ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "6548355e-7182-4d37-8673-944613f25755", "date": "Mon, 17 Mar 2025 16:55:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
86112025-03-17T16:55:07.030ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
86122025-03-17T16:55:07.032ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "864e6f95-30fa-4639-bb8f-c2bc86d55e52", "date": "Mon, 17 Mar 2025 16:55:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
86132025-03-17T16:55:07.032ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
86142025-03-17T16:55:07.033ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
86152025-03-17T16:55:07.033ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 441.116958ms iteration = 17 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
86162025-03-17T16:55:07.074ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86172025-03-17T16:55:07.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86182025-03-17T16:55:07.077ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
86192025-03-17T16:55:07.077ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86202025-03-17T16:55:07.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86212025-03-17T16:55:07.088ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
86222025-03-17T16:55:07.088ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f0ed0c-dcc8-4ba6-9d13-4163a5b042c4", "content-length": "134", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
86232025-03-17T16:55:07.088ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
86242025-03-17T16:55:07.088ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
86252025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
86262025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
86272025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86282025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86292025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86302025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
86312025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
86322025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
86332025-03-17T16:55:07.089ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
86342025-03-17T16:55:07.089ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86352025-03-17T16:55:07.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86362025-03-17T16:55:07.091ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
86372025-03-17T16:55:07.092ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd5bc9cc-546c-4495-9b21-1a413575de38", "content-length": "2", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
86382025-03-17T16:55:07.092ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
86392025-03-17T16:55:07.092ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "824e25cb-e0c0-4ef7-b02e-1c7bac2bf72f", "content-length": "52", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
86402025-03-17T16:55:07.092ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
86412025-03-17T16:55:07.092ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
86422025-03-17T16:55:07.107ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
86432025-03-17T16:55:07.185ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "7eb0e46d-39e0-4f26-96ac-9d9a4b19b958", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
86442025-03-17T16:55:07.185ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
86452025-03-17T16:55:07.185ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
86462025-03-17T16:55:07.186ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ced926c-1a23-4fa0-ad8a-1ea595b0b597", "content-length": "691", "date": "Mon, 17 Mar 2025 16:55:06 GMT"} })
86472025-03-17T16:55:07.186ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
86482025-03-17T16:55:07.186ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
86492025-03-17T16:55:07.186ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
86502025-03-17T16:55:07.186ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1000661a-0cef-4d7a-84b2-d245a2cad10f", "content-length": "682", "date": "Mon, 17 Mar 2025 16:55:07 GMT"} })
86512025-03-17T16:55:07.186ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86522025-03-17T16:55:07.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86532025-03-17T16:55:07.187ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 593.767997ms iteration = 29 status = {}
86542025-03-17T16:55:07.190ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86552025-03-17T16:55:07.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86562025-03-17T16:55:07.193ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86572025-03-17T16:55:07.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86582025-03-17T16:55:07.195ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86592025-03-17T16:55:07.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86602025-03-17T16:55:07.203ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 609.589025ms iteration = 29 status = {}
86612025-03-17T16:55:10.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 391f7372-f295-435c-85ec-0afd6066d0b7 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
86622025-03-17T16:55:16.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 86 reason = Timeout
86632025-03-17T16:55:16.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86642025-03-17T16:55:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86652025-03-17T16:55:16.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 86 reason = Timeout
86662025-03-17T16:55:16.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
86672025-03-17T16:55:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86682025-03-17T16:55:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86692025-03-17T16:55:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
86702025-03-17T16:55:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86712025-03-17T16:55:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86722025-03-17T16:55:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86732025-03-17T16:55:16.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86742025-03-17T16:55:16.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86752025-03-17T16:55:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
86762025-03-17T16:55:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86772025-03-17T16:55:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86782025-03-17T16:55:16.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
86792025-03-17T16:55:16.616ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.492213ms iteration = 86 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
86802025-03-17T16:55:16.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
86812025-03-17T16:55:16.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86822025-03-17T16:55:16.652ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.416382ms iteration = 86 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
86832025-03-17T16:55:20.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = c0e03f33-5c4c-4649-93a3-0fc38091ee1d response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
86842025-03-17T16:55:26.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 87 reason = Timeout
86852025-03-17T16:55:26.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86862025-03-17T16:55:26.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86872025-03-17T16:55:26.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 87 reason = Timeout
86882025-03-17T16:55:26.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
86892025-03-17T16:55:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86902025-03-17T16:55:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86912025-03-17T16:55:26.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
86922025-03-17T16:55:26.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86932025-03-17T16:55:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86942025-03-17T16:55:26.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86952025-03-17T16:55:26.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86962025-03-17T16:55:26.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86972025-03-17T16:55:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
86982025-03-17T16:55:26.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86992025-03-17T16:55:26.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87002025-03-17T16:55:26.614ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
87012025-03-17T16:55:26.614ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.39554ms iteration = 87 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
87022025-03-17T16:55:26.623ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87032025-03-17T16:55:26.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
87042025-03-17T16:55:26.625ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.024826ms iteration = 87 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
87052025-03-17T16:55:30.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 512 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 5d044ed5-14da-4d6d-aae3-47bd6b4cd511 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
87062025-03-17T16:55:35.124ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = 5a0e0de4-37ad-4601-8ddc-8089c3546d65 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
87072025-03-17T16:55:35.124ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = 5a0e0de4-37ad-4601-8ddc-8089c3546d65 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
87082025-03-17T16:55:35.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3440 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = 5a0e0de4-37ad-4601-8ddc-8089c3546d65 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
87092025-03-17T16:55:35.128ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = 09f6c3c0-f6f8-440f-84ab-582c2cdb4234 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
87102025-03-17T16:55:35.128ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = 09f6c3c0-f6f8-440f-84ab-582c2cdb4234 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
87112025-03-17T16:55:35.130ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2296 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62656 req_id = 09f6c3c0-f6f8-440f-84ab-582c2cdb4234 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
87122025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 30 reason = Timeout
87132025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 88 reason = Timeout
87142025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
87152025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87162025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 30 reason = Timeout
87172025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 30 reason = Timeout
87182025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
87192025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
87202025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87212025-03-17T16:55:36.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
87222025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 88 reason = Timeout
87232025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 32 reason = Timeout
87242025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87252025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87262025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 30 reason = Timeout
87272025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
87282025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 30 reason = Timeout
87292025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
87302025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 30 reason = Timeout
87312025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
87322025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 30 reason = Timeout
87332025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
87342025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 30 reason = Timeout
87352025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87362025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 30 reason = Timeout
87372025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 30 reason = Timeout
87382025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87392025-03-17T16:55:36.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
87402025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87412025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 30 reason = Timeout
87422025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
87432025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
87442025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 30 reason = Timeout
87452025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
87462025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 30 reason = Timeout
87472025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87482025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 30 reason = Timeout
87492025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87502025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87512025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
87522025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
87532025-03-17T16:55:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
87542025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
87552025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
87562025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87572025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87582025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87592025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
87602025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
87612025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
87622025-03-17T16:55:36.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
87632025-03-17T16:55:36.595ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4623, tv_nsec: 858159049 } })
87642025-03-17T16:55:36.595ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
87652025-03-17T16:55:36.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87662025-03-17T16:55:36.596ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87672025-03-17T16:55:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4623, tv_nsec: 859615460 } })
87682025-03-17T16:55:36.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
87692025-03-17T16:55:36.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.829844ms iteration = 30 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
87702025-03-17T16:55:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
87712025-03-17T16:55:36.599ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87722025-03-17T16:55:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
87732025-03-17T16:55:36.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87742025-03-17T16:55:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87752025-03-17T16:55:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87762025-03-17T16:55:36.602ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87772025-03-17T16:55:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
87782025-03-17T16:55:36.605ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87792025-03-17T16:55:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87802025-03-17T16:55:36.605ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
87812025-03-17T16:55:36.605ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
87822025-03-17T16:55:36.605ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
87832025-03-17T16:55:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
87842025-03-17T16:55:36.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87852025-03-17T16:55:36.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
87862025-03-17T16:55:36.609ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87872025-03-17T16:55:36.609ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87882025-03-17T16:55:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
87892025-03-17T16:55:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87902025-03-17T16:55:36.613ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87912025-03-17T16:55:36.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87922025-03-17T16:55:36.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 30 reason = Timeout
87932025-03-17T16:55:36.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 30 reason = Timeout
87942025-03-17T16:55:36.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
87952025-03-17T16:55:36.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
87962025-03-17T16:55:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
87972025-03-17T16:55:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
87982025-03-17T16:55:36.642ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
87992025-03-17T16:55:36.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88002025-03-17T16:55:36.643ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 49.915669ms iteration = 30 status = {"errors":[],"garbage_collect_requested":[]}
88012025-03-17T16:55:36.643ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
88022025-03-17T16:55:36.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
88032025-03-17T16:55:36.645ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88042025-03-17T16:55:36.648ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88052025-03-17T16:55:36.648ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
88062025-03-17T16:55:36.648ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88072025-03-17T16:55:36.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88082025-03-17T16:55:36.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88092025-03-17T16:55:36.648ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88102025-03-17T16:55:36.650ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88112025-03-17T16:55:36.650ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88122025-03-17T16:55:36.652ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.337621ms iteration = 88 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
88132025-03-17T16:55:36.653ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
88142025-03-17T16:55:36.653ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
88152025-03-17T16:55:36.653ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88162025-03-17T16:55:36.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
88172025-03-17T16:55:36.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88182025-03-17T16:55:36.655ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88192025-03-17T16:55:36.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88202025-03-17T16:55:36.656ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88212025-03-17T16:55:36.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88222025-03-17T16:55:36.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88232025-03-17T16:55:36.657ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
88242025-03-17T16:55:36.657ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
88252025-03-17T16:55:36.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
88262025-03-17T16:55:36.659ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
88272025-03-17T16:55:36.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88282025-03-17T16:55:36.659ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 43.657385ms iteration = 30 status = {"errors":[],"requests_created_ok":[]}
88292025-03-17T16:55:36.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88302025-03-17T16:55:36.659ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88312025-03-17T16:55:36.659ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
88322025-03-17T16:55:36.659ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
88332025-03-17T16:55:36.660ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88342025-03-17T16:55:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88352025-03-17T16:55:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
88362025-03-17T16:55:36.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88372025-03-17T16:55:36.660ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4623, tv_nsec: 923731734 } })
88382025-03-17T16:55:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88392025-03-17T16:55:36.661ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
88402025-03-17T16:55:36.661ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88412025-03-17T16:55:36.661ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88422025-03-17T16:55:36.662ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 46.740991ms iteration = 30 status = {"errors":[],"finish_invoked_ok":[]}
88432025-03-17T16:55:36.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
88442025-03-17T16:55:36.662ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88452025-03-17T16:55:36.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88462025-03-17T16:55:36.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
88472025-03-17T16:55:36.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
88482025-03-17T16:55:36.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88492025-03-17T16:55:36.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
88502025-03-17T16:55:36.667ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88512025-03-17T16:55:36.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88522025-03-17T16:55:36.669ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 76.12298ms iteration = 30 status = 0
88532025-03-17T16:55:36.671ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
88542025-03-17T16:55:36.671ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88552025-03-17T16:55:36.671ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b40c9d0-a96b-40e0-9e5c-6330856bcdc1", "content-length": "26", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
88562025-03-17T16:55:36.671ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88572025-03-17T16:55:36.674ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
88582025-03-17T16:55:36.674ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88592025-03-17T16:55:36.674ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "167869a8-b4d4-4e30-b7f2-8eebe2bfc5c7", "content-length": "26", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
88602025-03-17T16:55:36.674ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88612025-03-17T16:55:36.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
88622025-03-17T16:55:36.677ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
88632025-03-17T16:55:36.683ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 90.380175ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
88642025-03-17T16:55:36.685ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
88652025-03-17T16:55:36.685ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 93.171165ms iteration = 88 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
88662025-03-17T16:55:36.686ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
88672025-03-17T16:55:36.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88682025-03-17T16:55:36.691ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 98.003049ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
88692025-03-17T16:55:36.698ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88702025-03-17T16:55:36.698ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
88712025-03-17T16:55:36.698ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 104.806255ms iteration = 30 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
88722025-03-17T16:55:36.713ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88732025-03-17T16:55:36.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88742025-03-17T16:55:36.715ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88752025-03-17T16:55:36.716ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
88762025-03-17T16:55:36.716ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 123.137037ms iteration = 30 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
88772025-03-17T16:55:36.717ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
88782025-03-17T16:55:36.717ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88792025-03-17T16:55:36.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88802025-03-17T16:55:36.718ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 124.777953ms iteration = 30 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
88812025-03-17T16:55:36.734ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88822025-03-17T16:55:36.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88832025-03-17T16:55:36.736ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
88842025-03-17T16:55:36.736ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88852025-03-17T16:55:36.736ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec2e6a72-977d-4292-8bb8-c88493d82b5c", "content-length": "26", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
88862025-03-17T16:55:36.736ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88872025-03-17T16:55:36.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88882025-03-17T16:55:36.739ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
88892025-03-17T16:55:36.740ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
88902025-03-17T16:55:36.740ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88912025-03-17T16:55:36.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88922025-03-17T16:55:36.741ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88932025-03-17T16:55:36.743ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
88942025-03-17T16:55:36.743ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88952025-03-17T16:55:36.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88962025-03-17T16:55:36.749ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
88972025-03-17T16:55:36.749ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 156.158695ms iteration = 30 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
88982025-03-17T16:55:36.759ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
88992025-03-17T16:55:36.760ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce21426a-1a12-41e3-9e51-ff04287fdf1f", "content-length": "2", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89002025-03-17T16:55:36.761ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
89012025-03-17T16:55:36.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
89022025-03-17T16:55:36.765ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 173.647599ms iteration = 30 status = {}
89032025-03-17T16:55:36.766ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 173.644819ms iteration = 32 status = {"physical_disks_added":0}
89042025-03-17T16:55:36.818ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
89052025-03-17T16:55:36.819ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89062025-03-17T16:55:36.819ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c1f7d13-4022-4613-9871-446a3f44a580", "content-length": "372", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89072025-03-17T16:55:36.819ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89082025-03-17T16:55:36.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89092025-03-17T16:55:36.821ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 229.21885ms iteration = 30 status = null
89102025-03-17T16:55:36.847ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
89112025-03-17T16:55:36.848ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fce91eda-5242-401d-8168-24372b6b70d0", "content-length": "691", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89122025-03-17T16:55:36.848ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
89132025-03-17T16:55:36.848ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
89142025-03-17T16:55:36.848ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
89152025-03-17T16:55:36.850ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 257.27042ms iteration = 30 status = {}
89162025-03-17T16:55:36.866ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
89172025-03-17T16:55:36.866ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
89182025-03-17T16:55:36.866ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a1b64be5-c5aa-4c46-beca-3e7992a47d92", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89192025-03-17T16:55:36.866ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89202025-03-17T16:55:36.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89212025-03-17T16:55:36.869ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89222025-03-17T16:55:36.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89232025-03-17T16:55:36.872ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89242025-03-17T16:55:36.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89252025-03-17T16:55:36.876ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
89262025-03-17T16:55:36.876ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89272025-03-17T16:55:36.876ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "6c9dcdc3-9bf7-4134-83fa-bb85fa149840", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89282025-03-17T16:55:36.878ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 284.882652ms iteration = 30 status = {}
89292025-03-17T16:55:36.887ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89302025-03-17T16:55:36.888ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89312025-03-17T16:55:36.890ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
89322025-03-17T16:55:36.890ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89332025-03-17T16:55:36.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89342025-03-17T16:55:36.897ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
89352025-03-17T16:55:36.897ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "165eb992-4a70-472c-a9b4-916f353b8f41", "content-length": "134", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89362025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
89372025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
89382025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
89392025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
89402025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89412025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89422025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89432025-03-17T16:55:36.897ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
89442025-03-17T16:55:36.898ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
89452025-03-17T16:55:36.898ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
89462025-03-17T16:55:36.898ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
89472025-03-17T16:55:36.898ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89482025-03-17T16:55:36.898ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89492025-03-17T16:55:36.900ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
89502025-03-17T16:55:36.900ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "47b5423d-d7e0-43d9-a602-c9d2f37d9074", "content-length": "2", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89512025-03-17T16:55:36.900ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
89522025-03-17T16:55:36.900ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab0f6234-9236-473b-8ba7-509fabfb19fc", "content-length": "52", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89532025-03-17T16:55:36.900ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
89542025-03-17T16:55:36.900ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
89552025-03-17T16:55:36.992ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6669e0e8-1492-40b1-907f-a8cc9f18658d", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89562025-03-17T16:55:36.992ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
89572025-03-17T16:55:36.992ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
89582025-03-17T16:55:36.992ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d73efeb-34bc-46a5-9e7f-4a25e0bb2ff3", "content-length": "682", "date": "Mon, 17 Mar 2025 16:55:36 GMT"} })
89592025-03-17T16:55:36.992ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89602025-03-17T16:55:36.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89612025-03-17T16:55:36.996ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89622025-03-17T16:55:36.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89632025-03-17T16:55:36.999ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89642025-03-17T16:55:36.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89652025-03-17T16:55:37.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89662025-03-17T16:55:37.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
89672025-03-17T16:55:37.009ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 416.350097ms iteration = 30 status = {}
89682025-03-17T16:55:40.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 333 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 22f2bca6-d4dd-4816-beba-0e51c64fd31b response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
89692025-03-17T16:55:46.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 89 reason = Timeout
89702025-03-17T16:55:46.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89712025-03-17T16:55:46.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89722025-03-17T16:55:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 89 reason = Timeout
89732025-03-17T16:55:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89742025-03-17T16:55:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89752025-03-17T16:55:46.594ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89762025-03-17T16:55:46.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89772025-03-17T16:55:46.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89782025-03-17T16:55:46.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
89792025-03-17T16:55:46.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89802025-03-17T16:55:46.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89812025-03-17T16:55:46.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89822025-03-17T16:55:46.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89832025-03-17T16:55:46.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89842025-03-17T16:55:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
89852025-03-17T16:55:46.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89862025-03-17T16:55:46.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89872025-03-17T16:55:46.613ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
89882025-03-17T16:55:46.613ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.772294ms iteration = 89 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
89892025-03-17T16:55:46.625ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.356482ms iteration = 89 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
89902025-03-17T16:55:50.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = aa6fc0c3-cd99-46f1-a64b-5ac0421808df response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
89912025-03-17T16:55:56.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 90 reason = Timeout
89922025-03-17T16:55:56.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89932025-03-17T16:55:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89942025-03-17T16:55:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 90 reason = Timeout
89952025-03-17T16:55:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89962025-03-17T16:55:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89972025-03-17T16:55:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89982025-03-17T16:55:56.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89992025-03-17T16:55:56.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90002025-03-17T16:55:56.597ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.444365ms iteration = 90 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90012025-03-17T16:55:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
90022025-03-17T16:55:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90032025-03-17T16:55:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90042025-03-17T16:55:56.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90052025-03-17T16:55:56.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90062025-03-17T16:55:56.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90072025-03-17T16:55:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
90082025-03-17T16:55:56.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90092025-03-17T16:55:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90102025-03-17T16:55:56.617ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
90112025-03-17T16:55:56.617ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.021466ms iteration = 90 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
90122025-03-17T16:56:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 390 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = d6f478d8-66b2-48b7-bc05-e4d8589f4c97 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
90132025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 18 reason = Timeout
90142025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 16 reason = Timeout
90152025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
90162025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 31 reason = Timeout
90172025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
90182025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
90192025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 16 reason = Timeout
90202025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 18 reason = Timeout
90212025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
90222025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 16 reason = Timeout
90232025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
90242025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
90252025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 91 reason = Timeout
90262025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90272025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 16 reason = Timeout
90282025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 16 reason = Timeout
90292025-03-17T16:56:06.592ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:46:06.592353400 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
90302025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 16 reason = Timeout
90312025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
90322025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 31 reason = Timeout
90332025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
90342025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 31 reason = Timeout
90352025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
90362025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90372025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
90382025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
90392025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
90402025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
90412025-03-17T16:56:06.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
90422025-03-17T16:56:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4653, tv_nsec: 856025240 } })
90432025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 91 reason = Timeout
90442025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90452025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 33 reason = Timeout
90462025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90472025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 31 reason = Timeout
90482025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
90492025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 31 reason = Timeout
90502025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 31 reason = Timeout
90512025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
90522025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90532025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 31 reason = Timeout
90542025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 7 reason = Timeout
90552025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
90562025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 17 reason = Timeout
90572025-03-17T16:56:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
90582025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
90592025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.31\u{b5}s iteration = 17 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
90602025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 16 reason = Timeout
90612025-03-17T16:56:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
90622025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 31 reason = Timeout
90632025-03-17T16:56:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4653, tv_nsec: 856307326 } })
90642025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
90652025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 31 reason = Timeout
90662025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 16 reason = Timeout
90672025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
90682025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90692025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
90702025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
90712025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 6 reason = Timeout
90722025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
90732025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 31 reason = Timeout
90742025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
90752025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 31 reason = Timeout
90762025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 31 reason = Timeout
90772025-03-17T16:56:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
90782025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90792025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90802025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 31 reason = Timeout
90812025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 4 reason = Timeout
90822025-03-17T16:56:06.593ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
90832025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
90842025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
90852025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 16 reason = Timeout
90862025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
90872025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 16 reason = Timeout
90882025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
90892025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 31 reason = Timeout
90902025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
90912025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90922025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90932025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90942025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
90952025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
90962025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
90972025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
90982025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
90992025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
91002025-03-17T16:56:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91012025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91022025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
91032025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
91042025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
91052025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.730817ms iteration = 16 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
91062025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
91072025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
91082025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
91092025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
91102025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.718877ms iteration = 16 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
91112025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91122025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
91132025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
91142025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
91152025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91162025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
91172025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
91182025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
91192025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
91202025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
91212025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
91222025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
91232025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
91242025-03-17T16:56:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
91252025-03-17T16:56:06.595ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
91262025-03-17T16:56:06.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
91272025-03-17T16:56:06.596ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91282025-03-17T16:56:06.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4653, tv_nsec: 859538425 } })
91292025-03-17T16:56:06.596ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
91302025-03-17T16:56:06.597ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.857084ms iteration = 31 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
91312025-03-17T16:56:06.598ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4653, tv_nsec: 861665470 } })
91322025-03-17T16:56:06.598ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
91332025-03-17T16:56:06.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
91342025-03-17T16:56:06.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91352025-03-17T16:56:06.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91362025-03-17T16:56:06.601ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:46:06.592353400 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
91372025-03-17T16:56:06.601ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.832229ms iteration = 16 status = {"expiration":"2025-03-17T16:46:06.592353400Z","pruned":[]}
91382025-03-17T16:56:06.601ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91392025-03-17T16:56:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
91402025-03-17T16:56:06.603ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
91412025-03-17T16:56:06.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
91422025-03-17T16:56:06.604ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
91432025-03-17T16:56:06.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
91442025-03-17T16:56:06.606ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91452025-03-17T16:56:06.607ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91462025-03-17T16:56:06.608ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91472025-03-17T16:56:06.609ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
91482025-03-17T16:56:06.609ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91492025-03-17T16:56:06.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91502025-03-17T16:56:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
91512025-03-17T16:56:06.610ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
91522025-03-17T16:56:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
91532025-03-17T16:56:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
91542025-03-17T16:56:06.611ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
91552025-03-17T16:56:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
91562025-03-17T16:56:06.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91572025-03-17T16:56:06.613ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
91582025-03-17T16:56:06.613ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
91592025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 31 reason = Timeout
91602025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 21.855208ms iteration = 16 status = {"errors":[],"found_port_ok":[]}
91612025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
91622025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 4 reason = Timeout
91632025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 31 reason = Timeout
91642025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
91652025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
91662025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
91672025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91682025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
91692025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
91702025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91712025-03-17T16:56:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
91722025-03-17T16:56:06.618ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91732025-03-17T16:56:06.619ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91742025-03-17T16:56:06.620ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
91752025-03-17T16:56:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
91762025-03-17T16:56:06.620ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
91772025-03-17T16:56:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
91782025-03-17T16:56:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
91792025-03-17T16:56:06.620ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91802025-03-17T16:56:06.623ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
91812025-03-17T16:56:06.624ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
91822025-03-17T16:56:06.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
91832025-03-17T16:56:06.625ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
91842025-03-17T16:56:06.628ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91852025-03-17T16:56:06.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91862025-03-17T16:56:06.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91872025-03-17T16:56:06.631ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
91882025-03-17T16:56:06.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
91892025-03-17T16:56:06.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
91902025-03-17T16:56:06.632ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
91912025-03-17T16:56:06.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91922025-03-17T16:56:06.634ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4653, tv_nsec: 897401576 } })
91932025-03-17T16:56:06.634ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
91942025-03-17T16:56:06.635ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
91952025-03-17T16:56:06.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
91962025-03-17T16:56:06.638ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91972025-03-17T16:56:06.646ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
91982025-03-17T16:56:06.646ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
91992025-03-17T16:56:06.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92002025-03-17T16:56:06.649ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
92012025-03-17T16:56:06.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
92022025-03-17T16:56:06.649ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 56.49568ms iteration = 31 status = {"errors":[],"garbage_collect_requested":[]}
92032025-03-17T16:56:06.650ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
92042025-03-17T16:56:06.651ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
92052025-03-17T16:56:06.651ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 36.158084ms iteration = 31 status = {"errors":[],"finish_invoked_ok":[]}
92062025-03-17T16:56:06.652ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92072025-03-17T16:56:06.653ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92082025-03-17T16:56:06.655ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
92092025-03-17T16:56:06.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
92102025-03-17T16:56:06.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
92112025-03-17T16:56:06.657ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92122025-03-17T16:56:06.657ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
92132025-03-17T16:56:06.657ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
92142025-03-17T16:56:06.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
92152025-03-17T16:56:06.659ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
92162025-03-17T16:56:06.659ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
92172025-03-17T16:56:06.659ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
92182025-03-17T16:56:06.660ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92192025-03-17T16:56:06.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
92202025-03-17T16:56:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
92212025-03-17T16:56:06.661ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
92222025-03-17T16:56:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
92232025-03-17T16:56:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
92242025-03-17T16:56:06.662ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
92252025-03-17T16:56:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
92262025-03-17T16:56:06.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92272025-03-17T16:56:06.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92282025-03-17T16:56:06.664ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
92292025-03-17T16:56:06.664ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
92302025-03-17T16:56:06.664ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 70.621852ms iteration = 16 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
92312025-03-17T16:56:06.665ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92322025-03-17T16:56:06.665ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92332025-03-17T16:56:06.666ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
92342025-03-17T16:56:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
92352025-03-17T16:56:06.666ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92362025-03-17T16:56:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92372025-03-17T16:56:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
92382025-03-17T16:56:06.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92392025-03-17T16:56:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92402025-03-17T16:56:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
92412025-03-17T16:56:06.667ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
92422025-03-17T16:56:06.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
92432025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
92442025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
92452025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
92462025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
92472025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
92482025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
92492025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
92502025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
92512025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
92522025-03-17T16:56:06.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
92532025-03-17T16:56:06.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
92542025-03-17T16:56:06.671ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92552025-03-17T16:56:06.672ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92562025-03-17T16:56:06.672ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
92572025-03-17T16:56:06.672ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
92582025-03-17T16:56:06.672ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
92592025-03-17T16:56:06.673ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
92602025-03-17T16:56:06.673ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
92612025-03-17T16:56:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
92622025-03-17T16:56:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
92632025-03-17T16:56:06.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
92642025-03-17T16:56:06.673ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92652025-03-17T16:56:06.675ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
92662025-03-17T16:56:06.675ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
92672025-03-17T16:56:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
92682025-03-17T16:56:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
92692025-03-17T16:56:06.675ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
92702025-03-17T16:56:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
92712025-03-17T16:56:06.676ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92722025-03-17T16:56:06.676ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 61.499037ms iteration = 31 status = {"errors":[],"requests_created_ok":[]}
92732025-03-17T16:56:06.677ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
92742025-03-17T16:56:06.677ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92752025-03-17T16:56:06.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92762025-03-17T16:56:06.678ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
92772025-03-17T16:56:06.678ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
92782025-03-17T16:56:06.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
92792025-03-17T16:56:06.680ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92802025-03-17T16:56:06.681ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 88.191669ms iteration = 31 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
92812025-03-17T16:56:06.681ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
92822025-03-17T16:56:06.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
92832025-03-17T16:56:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
92842025-03-17T16:56:06.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
92852025-03-17T16:56:06.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
92862025-03-17T16:56:06.684ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
92872025-03-17T16:56:06.684ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
92882025-03-17T16:56:06.684ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
92892025-03-17T16:56:06.684ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
92902025-03-17T16:56:06.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
92912025-03-17T16:56:06.686ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
92922025-03-17T16:56:06.686ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 93.999093ms iteration = 91 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
92932025-03-17T16:56:06.687ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
92942025-03-17T16:56:06.687ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
92952025-03-17T16:56:06.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
92962025-03-17T16:56:06.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
92972025-03-17T16:56:06.687ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92982025-03-17T16:56:06.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92992025-03-17T16:56:06.688ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 94.833551ms iteration = 7 status = {"nsuccess":5}
93002025-03-17T16:56:06.688ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93012025-03-17T16:56:06.689ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93022025-03-17T16:56:06.690ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93032025-03-17T16:56:06.692ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 98.909818ms iteration = 31 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
93042025-03-17T16:56:06.693ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
93052025-03-17T16:56:06.693ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
93062025-03-17T16:56:06.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
93072025-03-17T16:56:06.693ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
93082025-03-17T16:56:06.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
93092025-03-17T16:56:06.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
93102025-03-17T16:56:06.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
93112025-03-17T16:56:06.694ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93122025-03-17T16:56:06.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93132025-03-17T16:56:06.695ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93142025-03-17T16:56:06.695ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93152025-03-17T16:56:06.696ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
93162025-03-17T16:56:06.696ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93172025-03-17T16:56:06.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
93182025-03-17T16:56:06.696ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
93192025-03-17T16:56:06.696ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
93202025-03-17T16:56:06.696ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
93212025-03-17T16:56:06.696ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93222025-03-17T16:56:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
93232025-03-17T16:56:06.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93242025-03-17T16:56:06.698ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
93252025-03-17T16:56:06.698ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93262025-03-17T16:56:06.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
93272025-03-17T16:56:06.700ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
93282025-03-17T16:56:06.700ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
93292025-03-17T16:56:06.700ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
93302025-03-17T16:56:06.700ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
93312025-03-17T16:56:06.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
93322025-03-17T16:56:06.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
93332025-03-17T16:56:06.702ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93342025-03-17T16:56:06.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93352025-03-17T16:56:06.704ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
93362025-03-17T16:56:06.704ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 111.12222ms iteration = 31 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
93372025-03-17T16:56:06.704ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93382025-03-17T16:56:06.704ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
93392025-03-17T16:56:06.704ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
93402025-03-17T16:56:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
93412025-03-17T16:56:06.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
93422025-03-17T16:56:06.705ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 112.197033ms iteration = 91 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
93432025-03-17T16:56:06.707ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93442025-03-17T16:56:06.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93452025-03-17T16:56:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
93462025-03-17T16:56:06.711ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93472025-03-17T16:56:06.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93482025-03-17T16:56:06.713ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93492025-03-17T16:56:06.715ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 122.031174ms iteration = 31 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
93502025-03-17T16:56:06.716ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
93512025-03-17T16:56:06.716ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
93522025-03-17T16:56:06.716ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 122.694848ms iteration = 31 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
93532025-03-17T16:56:06.716ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93542025-03-17T16:56:06.716ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
93552025-03-17T16:56:06.716ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
93562025-03-17T16:56:06.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93572025-03-17T16:56:06.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
93582025-03-17T16:56:06.718ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
93592025-03-17T16:56:06.718ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
93602025-03-17T16:56:06.718ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 124.65961ms iteration = 16 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
93612025-03-17T16:56:06.718ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93622025-03-17T16:56:06.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93632025-03-17T16:56:06.718ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93642025-03-17T16:56:06.718ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93652025-03-17T16:56:06.721ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
93662025-03-17T16:56:06.721ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 127.562502ms iteration = 31 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
93672025-03-17T16:56:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
93682025-03-17T16:56:06.722ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93692025-03-17T16:56:06.722ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
93702025-03-17T16:56:06.722ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
93712025-03-17T16:56:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93722025-03-17T16:56:06.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
93732025-03-17T16:56:06.726ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93742025-03-17T16:56:06.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93752025-03-17T16:56:06.728ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 134.518921ms iteration = 31 status = 0
93762025-03-17T16:56:06.728ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
93772025-03-17T16:56:06.728ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 134.771347ms iteration = 6 status = {"blueprint_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","inventory_collection_id":"82d5f958-8fc6-485e-b53f-48cd0925649b","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
93782025-03-17T16:56:06.728ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93792025-03-17T16:56:06.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93802025-03-17T16:56:06.729ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
93812025-03-17T16:56:06.729ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
93822025-03-17T16:56:06.729ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
93832025-03-17T16:56:06.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
93842025-03-17T16:56:06.730ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93852025-03-17T16:56:06.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93862025-03-17T16:56:06.731ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93872025-03-17T16:56:06.732ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93882025-03-17T16:56:06.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93892025-03-17T16:56:06.735ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
93902025-03-17T16:56:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
93912025-03-17T16:56:06.735ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
93922025-03-17T16:56:06.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
93932025-03-17T16:56:06.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
93942025-03-17T16:56:06.740ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
93952025-03-17T16:56:06.740ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
93962025-03-17T16:56:06.740ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
93972025-03-17T16:56:06.740ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 148.064011ms iteration = 16 status = {"generation":2}
93982025-03-17T16:56:06.741ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93992025-03-17T16:56:06.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
94002025-03-17T16:56:06.745ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94012025-03-17T16:56:06.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94022025-03-17T16:56:06.747ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94032025-03-17T16:56:06.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94042025-03-17T16:56:06.749ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94052025-03-17T16:56:06.749ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94062025-03-17T16:56:06.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94072025-03-17T16:56:06.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
94082025-03-17T16:56:06.751ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
94092025-03-17T16:56:06.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
94102025-03-17T16:56:06.752ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94112025-03-17T16:56:06.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94122025-03-17T16:56:06.754ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
94132025-03-17T16:56:06.754ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
94142025-03-17T16:56:06.754ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
94152025-03-17T16:56:06.754ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 162.266866ms iteration = 16 status = {"generation":1}
94162025-03-17T16:56:06.763ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
94172025-03-17T16:56:06.763ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 170.826149ms iteration = 16 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
94182025-03-17T16:56:06.764ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94192025-03-17T16:56:06.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94202025-03-17T16:56:06.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
94212025-03-17T16:56:06.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
94222025-03-17T16:56:06.766ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94232025-03-17T16:56:06.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94242025-03-17T16:56:06.769ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
94252025-03-17T16:56:06.778ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
94262025-03-17T16:56:06.778ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
94272025-03-17T16:56:06.778ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "694f8d89-644c-4bbf-99b6-93ef97a19da1", "content-length": "26", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
94282025-03-17T16:56:06.778ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
94292025-03-17T16:56:06.780ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
94302025-03-17T16:56:06.785ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
94312025-03-17T16:56:06.785ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 191.835209ms iteration = 16 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
94322025-03-17T16:56:06.786ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
94332025-03-17T16:56:06.786ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
94342025-03-17T16:56:06.786ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "79d04ea0-5c0d-4572-8895-afe8316e3195", "content-length": "26", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
94352025-03-17T16:56:06.786ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
94362025-03-17T16:56:06.788ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
94372025-03-17T16:56:06.789ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 196.500289ms iteration = 33 status = {"physical_disks_added":0}
94382025-03-17T16:56:06.854ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
94392025-03-17T16:56:06.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
94402025-03-17T16:56:06.903ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
94412025-03-17T16:56:06.912ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
94422025-03-17T16:56:06.912ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
94432025-03-17T16:56:06.913ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fcb2108-2030-4746-9066-3e770791a39f", "content-length": "26", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
94442025-03-17T16:56:06.913ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
94452025-03-17T16:56:06.914ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
94462025-03-17T16:56:06.926ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
94472025-03-17T16:56:06.926ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
94482025-03-17T16:56:06.927ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3afd91a-fef7-49e3-8c83-2474760258d5", "content-length": "372", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
94492025-03-17T16:56:06.927ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94502025-03-17T16:56:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94512025-03-17T16:56:06.984ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
94522025-03-17T16:56:06.984ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "82180e92-363d-425b-a892-d096feaa537a", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
94532025-03-17T16:56:06.984ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
94542025-03-17T16:56:06.987ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "b3b743de-28c0-4fec-a7a7-7aa5e53f00c6", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
94552025-03-17T16:56:06.987ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
94562025-03-17T16:56:06.989ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
94572025-03-17T16:56:06.989ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 396.965822ms iteration = 18 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
94582025-03-17T16:56:06.999ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 407.366544ms iteration = 31 status = {}
94592025-03-17T16:56:07.032ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
94602025-03-17T16:56:07.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
94612025-03-17T16:56:07.034ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94622025-03-17T16:56:07.034ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94632025-03-17T16:56:07.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94642025-03-17T16:56:07.036ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
94652025-03-17T16:56:07.036ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
94662025-03-17T16:56:07.036ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
94672025-03-17T16:56:07.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
94682025-03-17T16:56:07.038ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94692025-03-17T16:56:07.038ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94702025-03-17T16:56:07.042ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
94712025-03-17T16:56:07.042ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94722025-03-17T16:56:07.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94732025-03-17T16:56:07.043ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
94742025-03-17T16:56:07.043ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
94752025-03-17T16:56:07.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
94762025-03-17T16:56:07.044ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94772025-03-17T16:56:07.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94782025-03-17T16:56:07.046ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94792025-03-17T16:56:07.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94802025-03-17T16:56:07.047ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94812025-03-17T16:56:07.051ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
94822025-03-17T16:56:07.051ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94832025-03-17T16:56:07.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94842025-03-17T16:56:07.056ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94852025-03-17T16:56:07.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94862025-03-17T16:56:07.057ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94872025-03-17T16:56:07.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94882025-03-17T16:56:07.059ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94892025-03-17T16:56:07.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94902025-03-17T16:56:07.062ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94912025-03-17T16:56:07.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94922025-03-17T16:56:07.064ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94932025-03-17T16:56:07.064ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94942025-03-17T16:56:07.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94952025-03-17T16:56:07.065ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
94962025-03-17T16:56:07.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
94972025-03-17T16:56:07.066ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
94982025-03-17T16:56:07.066ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
94992025-03-17T16:56:07.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
95002025-03-17T16:56:07.068ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95012025-03-17T16:56:07.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95022025-03-17T16:56:07.070ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95032025-03-17T16:56:07.071ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
95042025-03-17T16:56:07.072ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6d3400-3069-4225-98d0-2b1c1c2fde3b", "content-length": "2", "date": "Mon, 17 Mar 2025 16:56:07 GMT"} })
95052025-03-17T16:56:07.072ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
95062025-03-17T16:56:07.073ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
95072025-03-17T16:56:07.073ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95082025-03-17T16:56:07.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95092025-03-17T16:56:07.074ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 459.265356ms iteration = 4 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":3,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"8007234b-9b03-4ff0-af97-c65524f74b0a","time":"2025-03-17T16:46:07.086835970Z"},{"operation":"list","target_sled":"8007234b-9b03-4ff0-af97-c65524f74b0a","time":"2025-03-17T16:51:07.043280519Z"},{"operation":"list","target_sled":"8007234b-9b03-4ff0-af97-c65524f74b0a","time":"2025-03-17T16:56:07.072271948Z"}]}
95102025-03-17T16:56:07.074ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
95112025-03-17T16:56:07.074ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
95122025-03-17T16:56:07.075ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e0d44e3e-2073-45a4-bd32-b3d6da18c4f5", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95132025-03-17T16:56:07.075ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95142025-03-17T16:56:07.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95152025-03-17T16:56:07.075ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95162025-03-17T16:56:07.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95172025-03-17T16:56:07.076ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95182025-03-17T16:56:07.077ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95192025-03-17T16:56:07.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95202025-03-17T16:56:07.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95212025-03-17T16:56:07.078ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95222025-03-17T16:56:07.078ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
95232025-03-17T16:56:07.078ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e049ba0a-ba71-4098-815d-c7788cf3aad1", "content-length": "2", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95242025-03-17T16:56:07.080ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95252025-03-17T16:56:07.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95262025-03-17T16:56:07.080ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
95272025-03-17T16:56:07.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
95282025-03-17T16:56:07.081ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
95292025-03-17T16:56:07.081ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95302025-03-17T16:56:07.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95312025-03-17T16:56:07.082ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 490.422554ms iteration = 31 status = null
95322025-03-17T16:56:07.083ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
95332025-03-17T16:56:07.083ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
95342025-03-17T16:56:07.084ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "03bc0076-c4bb-4ca5-b0d5-1fcf6e2236cc", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95352025-03-17T16:56:07.086ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 492.942718ms iteration = 31 status = {}
95362025-03-17T16:56:07.087ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 4bbf4949-9681-44f4-9ce1-d34c7cfeaf8e, kind: Service { id: 3827727f-e72b-46db-972f-ef987798c7fe }, name: Name("nexus-3827727f-e72b-46db-972f-ef987798c7fe"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 212, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 64b2110d-a2ad-4bda-9407-3a44fd83a7ef, kind: Service { id: 807f84c2-ce3d-4190-8b1c-85a3b322601f }, name: Name("nexus-807f84c2-ce3d-4190-8b1c-85a3b322601f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c1e71a59-6914-4de0-a06a-4a40418bbcd0, kind: Service { id: b8f58801-e3d7-4a1a-b665-e14ca1b06822 }, name: Name("nexus-b8f58801-e3d7-4a1a-b665-e14ca1b06822"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 130])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 524f6c4e-dd14-4112-b941-145bff1dbf2b, kind: Service { id: 36bd5b8d-4179-4c7d-acfe-dc22937af04b }, name: Name("external-dns-36bd5b8d-4179-4c7d-acfe-dc22937af04b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 238])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 85143d5f-0adb-4664-a2ac-73adcf707df0, kind: Service { id: 91530a96-1dfe-4db2-b8a3-2a09e4675f90 }, name: Name("external-dns-91530a96-1dfe-4db2-b8a3-2a09e4675f90"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 26])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
95372025-03-17T16:56:07.087ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95382025-03-17T16:56:07.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95392025-03-17T16:56:07.089ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95402025-03-17T16:56:07.091ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
95412025-03-17T16:56:07.091ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
95422025-03-17T16:56:07.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
95432025-03-17T16:56:07.108ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
95442025-03-17T16:56:07.108ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "389e0519-ebbe-408f-8f1d-2e940bfb324a", "date": "Mon, 17 Mar 2025 16:56:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95452025-03-17T16:56:07.108ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95462025-03-17T16:56:07.110ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "1edbf05b-b7e7-4b31-857f-f782fedaab23", "date": "Mon, 17 Mar 2025 16:56:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95472025-03-17T16:56:07.111ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95482025-03-17T16:56:07.112ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1fa5f5e3-2c99-4392-b061-b751017af997", "date": "Mon, 17 Mar 2025 16:56:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95492025-03-17T16:56:07.112ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95502025-03-17T16:56:07.114ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95512025-03-17T16:56:07.114ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 522.393098ms iteration = 18 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
95522025-03-17T16:56:07.132ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95532025-03-17T16:56:07.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95542025-03-17T16:56:07.135ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
95552025-03-17T16:56:07.135ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95562025-03-17T16:56:07.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95572025-03-17T16:56:07.141ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
95582025-03-17T16:56:07.141ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9061cd70-7639-4a42-9026-0b8eedd9e40e", "content-length": "691", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95592025-03-17T16:56:07.141ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
95602025-03-17T16:56:07.141ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
95612025-03-17T16:56:07.141ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
95622025-03-17T16:56:07.142ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 549.281954ms iteration = 31 status = {}
95632025-03-17T16:56:07.145ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
95642025-03-17T16:56:07.146ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea6b49b5-3b6e-49e4-a53e-1bf6854cd375", "content-length": "134", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95652025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
95662025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
95672025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
95682025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
95692025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95702025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95712025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95722025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
95732025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
95742025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
95752025-03-17T16:56:07.146ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
95762025-03-17T16:56:07.146ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95772025-03-17T16:56:07.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95782025-03-17T16:56:07.148ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
95792025-03-17T16:56:07.148ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd4da46b-3660-475d-a64d-11b03f07aef0", "content-length": "2", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95802025-03-17T16:56:07.148ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
95812025-03-17T16:56:07.148ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cfb3fff-2f89-44f9-ab5d-c48044b5be19", "content-length": "52", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95822025-03-17T16:56:07.148ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
95832025-03-17T16:56:07.148ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
95842025-03-17T16:56:07.198ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
95852025-03-17T16:56:07.198ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 524f6c4e-dd14-4112-b941-145bff1dbf2b, kind: Service { id: 36bd5b8d-4179-4c7d-acfe-dc22937af04b }, name: Name("external-dns-36bd5b8d-4179-4c7d-acfe-dc22937af04b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 238])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 85143d5f-0adb-4664-a2ac-73adcf707df0, kind: Service { id: 91530a96-1dfe-4db2-b8a3-2a09e4675f90 }, name: Name("external-dns-91530a96-1dfe-4db2-b8a3-2a09e4675f90"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 26])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4bbf4949-9681-44f4-9ce1-d34c7cfeaf8e, kind: Service { id: 3827727f-e72b-46db-972f-ef987798c7fe }, name: Name("nexus-3827727f-e72b-46db-972f-ef987798c7fe"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 212, 246])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 64b2110d-a2ad-4bda-9407-3a44fd83a7ef, kind: Service { id: 807f84c2-ce3d-4190-8b1c-85a3b322601f }, name: Name("nexus-807f84c2-ce3d-4190-8b1c-85a3b322601f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c1e71a59-6914-4de0-a06a-4a40418bbcd0, kind: Service { id: b8f58801-e3d7-4a1a-b665-e14ca1b06822 }, name: Name("nexus-b8f58801-e3d7-4a1a-b665-e14ca1b06822"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 130])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
95862025-03-17T16:56:07.198ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
95872025-03-17T16:56:07.247ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1b36531e-ed13-4786-a6a1-0a603abed113", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95882025-03-17T16:56:07.247ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
95892025-03-17T16:56:07.247ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
95902025-03-17T16:56:07.248ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b93a6ae7-2e57-434b-917a-b1d3c8d322d5", "content-length": "682", "date": "Mon, 17 Mar 2025 16:56:06 GMT"} })
95912025-03-17T16:56:07.248ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95922025-03-17T16:56:07.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95932025-03-17T16:56:07.252ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95942025-03-17T16:56:07.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95952025-03-17T16:56:07.257ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95962025-03-17T16:56:07.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95972025-03-17T16:56:07.259ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
95982025-03-17T16:56:07.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
95992025-03-17T16:56:07.267ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 673.79037ms iteration = 31 status = {}
96002025-03-17T16:56:07.321ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 8007234b-9b03-4ff0-af97-c65524f74b0a, time_created: 2025-03-17T16:41:47.612611Z, time_modified: 2025-03-17T16:41:47.612611Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::1:0, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
96012025-03-17T16:56:07.321ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
96022025-03-17T16:56:07.321ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96032025-03-17T16:56:07.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96042025-03-17T16:56:07.324ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
96052025-03-17T16:56:07.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
96062025-03-17T16:56:07.326ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96072025-03-17T16:56:07.329ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
96082025-03-17T16:56:07.360ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
96092025-03-17T16:56:07.360ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "8fae11c2-895f-4eae-be1e-e66cd52b89a1", "date": "Mon, 17 Mar 2025 16:56:07 GMT"} })
96102025-03-17T16:56:07.363ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
96112025-03-17T16:56:07.363ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 769.885018ms
96122025-03-17T16:56:07.363ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 769.897218ms iteration = 4 status = {}
96132025-03-17T16:56:10.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 288 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 6a436fde-6a54-48f0-a1ba-5c752514d777 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
96142025-03-17T16:56:16.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 92 reason = Timeout
96152025-03-17T16:56:16.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96162025-03-17T16:56:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96172025-03-17T16:56:16.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 92 reason = Timeout
96182025-03-17T16:56:16.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96192025-03-17T16:56:16.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96202025-03-17T16:56:16.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96212025-03-17T16:56:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
96222025-03-17T16:56:16.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96232025-03-17T16:56:16.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96242025-03-17T16:56:16.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96252025-03-17T16:56:16.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96262025-03-17T16:56:16.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96272025-03-17T16:56:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
96282025-03-17T16:56:16.605ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96292025-03-17T16:56:16.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96302025-03-17T16:56:16.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
96312025-03-17T16:56:16.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.910271ms iteration = 92 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
96322025-03-17T16:56:16.624ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96332025-03-17T16:56:16.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96342025-03-17T16:56:16.653ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.324632ms iteration = 92 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
96352025-03-17T16:56:20.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 280 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 85ee442b-17ac-40ff-bd75-81544022c57f response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
96362025-03-17T16:56:26.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 93 reason = Timeout
96372025-03-17T16:56:26.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96382025-03-17T16:56:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96392025-03-17T16:56:26.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 93 reason = Timeout
96402025-03-17T16:56:26.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96412025-03-17T16:56:26.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96422025-03-17T16:56:26.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96432025-03-17T16:56:26.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96442025-03-17T16:56:26.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96452025-03-17T16:56:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
96462025-03-17T16:56:26.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96472025-03-17T16:56:26.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96482025-03-17T16:56:26.601ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96492025-03-17T16:56:26.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96502025-03-17T16:56:26.603ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96512025-03-17T16:56:26.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
96522025-03-17T16:56:26.606ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96532025-03-17T16:56:26.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96542025-03-17T16:56:26.615ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
96552025-03-17T16:56:26.615ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.713857ms iteration = 93 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
96562025-03-17T16:56:26.626ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.494037ms iteration = 93 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
96572025-03-17T16:56:30.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 295 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 729754e9-7da4-418c-8dfd-91f9c2313830 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
96582025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 32 reason = Timeout
96592025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 94 reason = Timeout
96602025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
96612025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96622025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 32 reason = Timeout
96632025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 32 reason = Timeout
96642025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
96652025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
96662025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96672025-03-17T16:56:36.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
96682025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 94 reason = Timeout
96692025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 34 reason = Timeout
96702025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96712025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 32 reason = Timeout
96722025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
96732025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
96742025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
96752025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 32 reason = Timeout
96762025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 32 reason = Timeout
96772025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
96782025-03-17T16:56:36.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
96792025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 32 reason = Timeout
96802025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
96812025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
96822025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 32 reason = Timeout
96832025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
96842025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 32 reason = Timeout
96852025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
96862025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 32 reason = Timeout
96872025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 32 reason = Timeout
96882025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
96892025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96902025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 32 reason = Timeout
96912025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 32 reason = Timeout
96922025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
96932025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
96942025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 32 reason = Timeout
96952025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
96962025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
96972025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
96982025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
96992025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97002025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
97012025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97022025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
97032025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97042025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
97052025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
97062025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97072025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
97082025-03-17T16:56:36.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
97092025-03-17T16:56:36.595ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4683, tv_nsec: 858494301 } })
97102025-03-17T16:56:36.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97112025-03-17T16:56:36.595ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
97122025-03-17T16:56:36.596ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.338263ms iteration = 32 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
97132025-03-17T16:56:36.596ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97142025-03-17T16:56:36.596ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4683, tv_nsec: 859992183 } })
97152025-03-17T16:56:36.597ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97162025-03-17T16:56:36.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
97172025-03-17T16:56:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
97182025-03-17T16:56:36.599ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97192025-03-17T16:56:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97202025-03-17T16:56:36.599ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
97212025-03-17T16:56:36.599ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97222025-03-17T16:56:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97232025-03-17T16:56:36.600ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
97242025-03-17T16:56:36.600ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
97252025-03-17T16:56:36.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
97262025-03-17T16:56:36.602ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97272025-03-17T16:56:36.602ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97282025-03-17T16:56:36.603ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97292025-03-17T16:56:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
97302025-03-17T16:56:36.604ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
97312025-03-17T16:56:36.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
97322025-03-17T16:56:36.605ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
97332025-03-17T16:56:36.605ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
97342025-03-17T16:56:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97352025-03-17T16:56:36.606ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
97362025-03-17T16:56:36.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97372025-03-17T16:56:36.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97382025-03-17T16:56:36.607ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97392025-03-17T16:56:36.609ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
97402025-03-17T16:56:36.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
97412025-03-17T16:56:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
97422025-03-17T16:56:36.610ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97432025-03-17T16:56:36.610ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97442025-03-17T16:56:36.610ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97452025-03-17T16:56:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97462025-03-17T16:56:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97472025-03-17T16:56:36.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97482025-03-17T16:56:36.612ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97492025-03-17T16:56:36.614ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
97502025-03-17T16:56:36.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
97512025-03-17T16:56:36.614ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4683, tv_nsec: 877513848 } })
97522025-03-17T16:56:36.614ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
97532025-03-17T16:56:36.614ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97542025-03-17T16:56:36.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97552025-03-17T16:56:36.614ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
97562025-03-17T16:56:36.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 32 reason = Timeout
97572025-03-17T16:56:36.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
97582025-03-17T16:56:36.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 32 reason = Timeout
97592025-03-17T16:56:36.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
97602025-03-17T16:56:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
97612025-03-17T16:56:36.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
97622025-03-17T16:56:36.636ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
97632025-03-17T16:56:36.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
97642025-03-17T16:56:36.638ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97652025-03-17T16:56:36.639ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 45.972975ms iteration = 32 status = {"errors":[],"garbage_collect_requested":[]}
97662025-03-17T16:56:36.639ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
97672025-03-17T16:56:36.639ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
97682025-03-17T16:56:36.639ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
97692025-03-17T16:56:36.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
97702025-03-17T16:56:36.640ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
97712025-03-17T16:56:36.640ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
97722025-03-17T16:56:36.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
97732025-03-17T16:56:36.642ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
97742025-03-17T16:56:36.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
97752025-03-17T16:56:36.648ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
97762025-03-17T16:56:36.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
97772025-03-17T16:56:36.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
97782025-03-17T16:56:36.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
97792025-03-17T16:56:36.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97802025-03-17T16:56:36.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97812025-03-17T16:56:36.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97822025-03-17T16:56:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
97832025-03-17T16:56:36.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97842025-03-17T16:56:36.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97852025-03-17T16:56:36.661ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
97862025-03-17T16:56:36.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97872025-03-17T16:56:36.661ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
97882025-03-17T16:56:36.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
97892025-03-17T16:56:36.663ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97902025-03-17T16:56:36.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
97912025-03-17T16:56:36.666ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97922025-03-17T16:56:36.666ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 51.653246ms iteration = 32 status = {"errors":[],"requests_created_ok":[]}
97932025-03-17T16:56:36.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97942025-03-17T16:56:36.668ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 53.384653ms iteration = 32 status = {"errors":[],"finish_invoked_ok":[]}
97952025-03-17T16:56:36.673ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
97962025-03-17T16:56:36.673ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 80.169647ms iteration = 32 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
97972025-03-17T16:56:36.673ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
97982025-03-17T16:56:36.673ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
97992025-03-17T16:56:36.674ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "deff5282-c135-429f-9775-86c5d1e2a3ca", "content-length": "26", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98002025-03-17T16:56:36.674ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
98012025-03-17T16:56:36.676ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
98022025-03-17T16:56:36.691ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
98032025-03-17T16:56:36.691ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
98042025-03-17T16:56:36.691ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bccbf8e-901e-4e98-886e-29ddd0ecc965", "content-length": "26", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98052025-03-17T16:56:36.691ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
98062025-03-17T16:56:36.693ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
98072025-03-17T16:56:36.806ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
98082025-03-17T16:56:36.806ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "5112f10c-0e25-4dd1-86bb-d8284bb5eb36", "content-length": "2", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98092025-03-17T16:56:36.808ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
98102025-03-17T16:56:36.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
98112025-03-17T16:56:36.822ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 229.838502ms iteration = 32 status = {}
98122025-03-17T16:56:36.839ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
98132025-03-17T16:56:36.840ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
98142025-03-17T16:56:36.840ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
98152025-03-17T16:56:36.840ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
98162025-03-17T16:56:36.840ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca48cf73-e025-4c19-aa26-b2408f4fa053", "content-length": "372", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98172025-03-17T16:56:36.840ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98182025-03-17T16:56:36.840ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe488bf-bdb5-46d2-89ca-d55cc50290fa", "content-length": "26", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98192025-03-17T16:56:36.840ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
98202025-03-17T16:56:36.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98212025-03-17T16:56:36.841ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
98222025-03-17T16:56:36.842ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98232025-03-17T16:56:36.843ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98242025-03-17T16:56:36.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98252025-03-17T16:56:36.844ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
98262025-03-17T16:56:36.844ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98272025-03-17T16:56:36.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98282025-03-17T16:56:36.848ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 256.053844ms iteration = 32 status = null
98292025-03-17T16:56:36.848ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98302025-03-17T16:56:36.851ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
98312025-03-17T16:56:36.851ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98322025-03-17T16:56:36.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98332025-03-17T16:56:36.853ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 260.178583ms iteration = 32 status = 0
98342025-03-17T16:56:36.854ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
98352025-03-17T16:56:36.854ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 262.046943ms iteration = 94 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
98362025-03-17T16:56:36.872ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 279.119828ms iteration = 94 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
98372025-03-17T16:56:36.872ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
98382025-03-17T16:56:36.872ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
98392025-03-17T16:56:36.872ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
98402025-03-17T16:56:36.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
98412025-03-17T16:56:36.872ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
98422025-03-17T16:56:36.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
98432025-03-17T16:56:36.873ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 280.006647ms iteration = 32 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
98442025-03-17T16:56:36.875ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98452025-03-17T16:56:36.875ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 282.272396ms iteration = 32 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
98462025-03-17T16:56:36.877ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
98472025-03-17T16:56:36.877ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
98482025-03-17T16:56:36.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
98492025-03-17T16:56:36.892ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
98502025-03-17T16:56:36.892ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
98512025-03-17T16:56:36.892ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "23cc838b-c44b-4a8b-8a67-996ca944cecb", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98522025-03-17T16:56:36.892ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98532025-03-17T16:56:36.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98542025-03-17T16:56:36.895ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98552025-03-17T16:56:36.895ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98562025-03-17T16:56:36.899ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98572025-03-17T16:56:36.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98582025-03-17T16:56:36.902ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
98592025-03-17T16:56:36.902ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
98602025-03-17T16:56:36.903ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4bed3ad1-09d8-437e-a40a-2c80fdb65df6", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98612025-03-17T16:56:36.903ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 310.641273ms iteration = 32 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
98622025-03-17T16:56:36.904ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 311.522442ms iteration = 32 status = {}
98632025-03-17T16:56:36.910ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
98642025-03-17T16:56:36.910ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
98652025-03-17T16:56:36.910ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 317.176973ms iteration = 32 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
98662025-03-17T16:56:36.913ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
98672025-03-17T16:56:36.913ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 319.737967ms iteration = 32 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
98682025-03-17T16:56:36.938ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 345.168142ms iteration = 34 status = {"physical_disks_added":0}
98692025-03-17T16:56:36.966ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98702025-03-17T16:56:36.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98712025-03-17T16:56:36.967ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
98722025-03-17T16:56:36.967ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "5478243f-eaaa-484d-a703-b14366e228ba", "content-length": "691", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98732025-03-17T16:56:36.967ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
98742025-03-17T16:56:36.967ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
98752025-03-17T16:56:36.967ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
98762025-03-17T16:56:36.968ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 375.128204ms iteration = 32 status = {}
98772025-03-17T16:56:36.968ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
98782025-03-17T16:56:36.968ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98792025-03-17T16:56:36.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98802025-03-17T16:56:36.979ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
98812025-03-17T16:56:36.979ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c36251-8d39-4efe-b23b-6f5c545b0d9a", "content-length": "134", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98822025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
98832025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
98842025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
98852025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
98862025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98872025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98882025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98892025-03-17T16:56:36.979ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
98902025-03-17T16:56:36.980ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
98912025-03-17T16:56:36.980ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
98922025-03-17T16:56:36.980ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
98932025-03-17T16:56:36.980ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98942025-03-17T16:56:36.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98952025-03-17T16:56:36.982ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
98962025-03-17T16:56:36.982ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "93ba1f26-8a21-4c29-909f-2784ee5f09a4", "content-length": "2", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98972025-03-17T16:56:36.982ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
98982025-03-17T16:56:36.982ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ac5803d-f360-47b3-bee8-b945ae911564", "content-length": "52", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
98992025-03-17T16:56:36.982ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
99002025-03-17T16:56:36.982ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
99012025-03-17T16:56:37.072ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "072ed671-e138-4415-a1dc-34a84c163cc5", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
99022025-03-17T16:56:37.072ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
99032025-03-17T16:56:37.072ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
99042025-03-17T16:56:37.072ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c6b907-f895-49ec-8273-693faba3f1ac", "content-length": "682", "date": "Mon, 17 Mar 2025 16:56:36 GMT"} })
99052025-03-17T16:56:37.072ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99062025-03-17T16:56:37.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99072025-03-17T16:56:37.076ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99082025-03-17T16:56:37.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99092025-03-17T16:56:37.079ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99102025-03-17T16:56:37.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99112025-03-17T16:56:37.081ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99122025-03-17T16:56:37.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99132025-03-17T16:56:37.090ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 496.704068ms iteration = 32 status = {}
99142025-03-17T16:56:40.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 70a02e52-7afe-4dac-8177-510d0a84a2d3 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
99152025-03-17T16:56:46.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 95 reason = Timeout
99162025-03-17T16:56:46.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99172025-03-17T16:56:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99182025-03-17T16:56:46.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 95 reason = Timeout
99192025-03-17T16:56:46.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99202025-03-17T16:56:46.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99212025-03-17T16:56:46.596ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99222025-03-17T16:56:46.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99232025-03-17T16:56:46.621ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99242025-03-17T16:56:46.624ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
99252025-03-17T16:56:46.625ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99262025-03-17T16:56:46.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99272025-03-17T16:56:46.625ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.20914ms iteration = 95 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
99282025-03-17T16:56:46.627ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99292025-03-17T16:56:46.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99302025-03-17T16:56:46.628ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99312025-03-17T16:56:46.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
99322025-03-17T16:56:46.631ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99332025-03-17T16:56:46.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99342025-03-17T16:56:46.640ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
99352025-03-17T16:56:46.640ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 47.151841ms iteration = 95 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
99362025-03-17T16:56:50.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 349 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = 93118031-f996-4b8d-852e-58d9f7dc06cc response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
99372025-03-17T16:56:56.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 96 reason = Timeout
99382025-03-17T16:56:56.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99392025-03-17T16:56:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99402025-03-17T16:56:56.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 96 reason = Timeout
99412025-03-17T16:56:56.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99422025-03-17T16:56:56.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99432025-03-17T16:56:56.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99442025-03-17T16:56:56.595ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99452025-03-17T16:56:56.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99462025-03-17T16:56:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
99472025-03-17T16:56:56.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99482025-03-17T16:56:56.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99492025-03-17T16:56:56.600ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99502025-03-17T16:56:56.600ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99512025-03-17T16:56:56.602ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99522025-03-17T16:56:56.604ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
99532025-03-17T16:56:56.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99542025-03-17T16:56:56.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99552025-03-17T16:56:56.613ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
99562025-03-17T16:56:56.613ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.490329ms iteration = 96 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
99572025-03-17T16:56:56.625ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.052316ms iteration = 96 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
99582025-03-17T16:57:00.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 271 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = fa6ac8d3-fd56-43cc-bbe7-2d4ada328a06 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f
99592025-03-17T16:57:05.123ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:46151
99602025-03-17T16:57:05.123ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46151 req_id = 0a861565-26b2-4266-ac4a-ad77f2dbb585 roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
99612025-03-17T16:57:05.124ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46151 req_id = 0a861565-26b2-4266-ac4a-ad77f2dbb585 resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
99622025-03-17T16:57:05.127ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3349 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46151 req_id = 0a861565-26b2-4266-ac4a-ad77f2dbb585 response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?sort_by=id_ascending
99632025-03-17T16:57:05.127ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46151 req_id = d14619bc-8741-47f6-af79-606b160d108b roles = RoleSet { roles: {} } uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
99642025-03-17T16:57:05.127ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46151 req_id = d14619bc-8741-47f6-af79-606b160d108b resource = Database result = Ok(()) uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
99652025-03-17T16:57:05.129ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2541 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:46151 req_id = d14619bc-8741-47f6-af79-606b160d108b response_code = 200 uri = /metrics/collectors/1374f180-03c9-4b69-abf3-a0c92428bb9a/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmY2RlNWY0OC1iMDE0LTQwNWEtOTA5Mi1jMTRiMzc5YTkyZjAifX0%3D
99662025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 19 reason = Timeout
99672025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 17 reason = Timeout
99682025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
99692025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
99702025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 97 reason = Timeout
99712025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 33 reason = Timeout
99722025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
99732025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99742025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 17 reason = Timeout
99752025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 33 reason = Timeout
99762025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
99772025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 19 reason = Timeout
99782025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 17 reason = Timeout
99792025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
99802025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
99812025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 17 reason = Timeout
99822025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 17 reason = Timeout
99832025-03-17T16:57:06.592ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-17 16:47:06.592518155 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
99842025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 17 reason = Timeout
99852025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
99862025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 33 reason = Timeout
99872025-03-17T16:57:06.592ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
99882025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
99892025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
99902025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99912025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
99922025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 17 reason = Timeout
99932025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 33 reason = Timeout
99942025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 18 reason = Timeout
99952025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
99962025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
99972025-03-17T16:57:06.593ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf
99982025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 33 reason = Timeout
99992025-03-17T16:57:06.593ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
100002025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 19.97\u{b5}s iteration = 18 status = {"enabled":false,"target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf"}
100012025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 33 reason = Timeout
100022025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 17 reason = Timeout
100032025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
100042025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100052025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
100062025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
100072025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 33 reason = Timeout
100082025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
100092025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 33 reason = Timeout
100102025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
100112025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 33 reason = Timeout
100122025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
100132025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 17 reason = Timeout
100142025-03-17T16:57:06.593ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
100152025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
100162025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 33 reason = Timeout
100172025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
100182025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 33 reason = Timeout
100192025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100202025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
100212025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 33 reason = Timeout
100222025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 35 reason = Timeout
100232025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
100242025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 17 reason = Timeout
100252025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
100262025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
100272025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 97 reason = Timeout
100282025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
100292025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100302025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 33 reason = Timeout
100312025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
100322025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 33 reason = Timeout
100332025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
100342025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
100352025-03-17T16:57:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("36bd5b8d-4179-4c7d-acfe-dc22937af04b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91530a96-1dfe-4db2-b8a3-2a09e4675f90.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 4713, tv_nsec: 856552710 } })
100362025-03-17T16:57:06.593ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3d7d62f4-103b-4ac4-8e49-636fa487c688.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ad3855ba-aca2-422e-ac2f-bfcceef64f67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("cdbf728e-4825-4a0b-a9ac-8878039a9fef.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 4713, tv_nsec: 856588060 } })
100372025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
100382025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100392025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
100402025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
100412025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
100422025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
100432025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
100442025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
100452025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100462025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
100472025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
100482025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100492025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
100502025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
100512025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
100522025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
100532025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
100542025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
100552025-03-17T16:57:06.593ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.35758ms iteration = 17 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
100562025-03-17T16:57:06.594ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
100572025-03-17T16:57:06.594ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.669196ms iteration = 17 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
100582025-03-17T16:57:06.594ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
100592025-03-17T16:57:06.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
100602025-03-17T16:57:06.597ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
100612025-03-17T16:57:06.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4713, tv_nsec: 860327471 } })
100622025-03-17T16:57:06.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
100632025-03-17T16:57:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
100642025-03-17T16:57:06.597ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4713, tv_nsec: 860796331 } })
100652025-03-17T16:57:06.597ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
100662025-03-17T16:57:06.599ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.682323ms iteration = 33 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
100672025-03-17T16:57:06.599ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-17 16:47:06.592518155 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
100682025-03-17T16:57:06.599ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 7.080242ms iteration = 17 status = {"expiration":"2025-03-17T16:47:06.592518155Z","pruned":[]}
100692025-03-17T16:57:06.600ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100702025-03-17T16:57:06.603ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
100712025-03-17T16:57:06.603ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100722025-03-17T16:57:06.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100732025-03-17T16:57:06.604ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 11.811122ms iteration = 17 status = {"errors":[],"found_port_ok":[]}
100742025-03-17T16:57:06.606ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100752025-03-17T16:57:06.606ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100762025-03-17T16:57:06.607ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100772025-03-17T16:57:06.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100782025-03-17T16:57:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
100792025-03-17T16:57:06.610ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
100802025-03-17T16:57:06.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
100812025-03-17T16:57:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
100822025-03-17T16:57:06.611ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
100832025-03-17T16:57:06.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
100842025-03-17T16:57:06.614ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100852025-03-17T16:57:06.615ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 33 reason = Timeout
100862025-03-17T16:57:06.615ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 33 reason = Timeout
100872025-03-17T16:57:06.615ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
100882025-03-17T16:57:06.615ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
100892025-03-17T16:57:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
100902025-03-17T16:57:06.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
100912025-03-17T16:57:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
100922025-03-17T16:57:06.616ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
100932025-03-17T16:57:06.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
100942025-03-17T16:57:06.618ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100952025-03-17T16:57:06.620ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
100962025-03-17T16:57:06.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
100972025-03-17T16:57:06.622ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
100982025-03-17T16:57:06.622ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100992025-03-17T16:57:06.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
101002025-03-17T16:57:06.626ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
101012025-03-17T16:57:06.626ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 33.757653ms iteration = 33 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
101022025-03-17T16:57:06.629ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101032025-03-17T16:57:06.632ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
101042025-03-17T16:57:06.632ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101052025-03-17T16:57:06.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101062025-03-17T16:57:06.636ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101072025-03-17T16:57:06.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101082025-03-17T16:57:06.639ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
101092025-03-17T16:57:06.640ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
101102025-03-17T16:57:06.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
101112025-03-17T16:57:06.640ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101122025-03-17T16:57:06.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
101132025-03-17T16:57:06.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101142025-03-17T16:57:06.644ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 50.89083ms iteration = 33 status = {"errors":[],"garbage_collect_requested":[]}
101152025-03-17T16:57:06.654ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101162025-03-17T16:57:06.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
101172025-03-17T16:57:06.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
101182025-03-17T16:57:06.656ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
101192025-03-17T16:57:06.656ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
101202025-03-17T16:57:06.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
101212025-03-17T16:57:06.657ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101222025-03-17T16:57:06.657ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101232025-03-17T16:57:06.658ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
101242025-03-17T16:57:06.658ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
101252025-03-17T16:57:06.658ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
101262025-03-17T16:57:06.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
101272025-03-17T16:57:06.658ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
101282025-03-17T16:57:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
101292025-03-17T16:57:06.659ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
101302025-03-17T16:57:06.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101312025-03-17T16:57:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
101322025-03-17T16:57:06.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101332025-03-17T16:57:06.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
101342025-03-17T16:57:06.662ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
101352025-03-17T16:57:06.662ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
101362025-03-17T16:57:06.662ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 68.835454ms iteration = 17 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
101372025-03-17T16:57:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
101382025-03-17T16:57:06.662ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
101392025-03-17T16:57:06.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
101402025-03-17T16:57:06.663ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101412025-03-17T16:57:06.664ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-8007234b-9b03-4ff0-af97-c65524f74b0a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 4713, tv_nsec: 927768425 } })
101422025-03-17T16:57:06.665ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
101432025-03-17T16:57:06.665ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101442025-03-17T16:57:06.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101452025-03-17T16:57:06.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
101462025-03-17T16:57:06.667ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
101472025-03-17T16:57:06.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
101482025-03-17T16:57:06.668ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 75.15867ms iteration = 97 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
101492025-03-17T16:57:06.669ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101502025-03-17T16:57:06.672ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
101512025-03-17T16:57:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
101522025-03-17T16:57:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: fdc99587-9a5b-42dc-a8d8-afea4b1691cf, lookup_type: ById(fdc99587-9a5b-42dc-a8d8-afea4b1691cf) } result = Ok(())
101532025-03-17T16:57:06.672ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101542025-03-17T16:57:06.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101552025-03-17T16:57:06.674ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 81.622178ms iteration = 33 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
101562025-03-17T16:57:06.675ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
101572025-03-17T16:57:06.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
101582025-03-17T16:57:06.677ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 62.57063ms iteration = 33 status = {"errors":[],"requests_created_ok":[]}
101592025-03-17T16:57:06.680ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf original_time_created = 2025-03-17 16:41:02.734916 UTC target_id = fdc99587-9a5b-42dc-a8d8-afea4b1691cf time_created = 2025-03-17 16:41:02.734916 UTC
101602025-03-17T16:57:06.680ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 88.499305ms iteration = 97 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"fdc99587-9a5b-42dc-a8d8-afea4b1691cf","time_created":"2025-03-17T16:41:02.734916Z"}
101612025-03-17T16:57:06.681ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101622025-03-17T16:57:06.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101632025-03-17T16:57:06.682ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101642025-03-17T16:57:06.684ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 69.162411ms iteration = 33 status = {"errors":[],"finish_invoked_ok":[]}
101652025-03-17T16:57:06.684ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
101662025-03-17T16:57:06.684ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101672025-03-17T16:57:06.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101682025-03-17T16:57:06.688ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
101692025-03-17T16:57:06.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
101702025-03-17T16:57:06.695ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
101712025-03-17T16:57:06.695ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
101722025-03-17T16:57:06.695ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
101732025-03-17T16:57:06.695ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
101742025-03-17T16:57:06.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
101752025-03-17T16:57:06.696ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
101762025-03-17T16:57:06.696ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
101772025-03-17T16:57:06.696ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
101782025-03-17T16:57:06.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
101792025-03-17T16:57:06.697ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 104.456227ms iteration = 33 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
101802025-03-17T16:57:06.697ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101812025-03-17T16:57:06.699ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
101822025-03-17T16:57:06.699ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
101832025-03-17T16:57:06.699ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 106.160914ms iteration = 33 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
101842025-03-17T16:57:06.699ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101852025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
101862025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
101872025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
101882025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
101892025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
101902025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
101912025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
101922025-03-17T16:57:06.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
101932025-03-17T16:57:06.702ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
101942025-03-17T16:57:06.702ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101952025-03-17T16:57:06.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101962025-03-17T16:57:06.703ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
101972025-03-17T16:57:06.703ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 0 version = 2
101982025-03-17T16:57:06.703ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
101992025-03-17T16:57:06.703ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102002025-03-17T16:57:06.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
102012025-03-17T16:57:06.703ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102022025-03-17T16:57:06.703ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
102032025-03-17T16:57:06.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
102042025-03-17T16:57:06.704ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102052025-03-17T16:57:06.705ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
102062025-03-17T16:57:06.705ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
102072025-03-17T16:57:06.705ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 111.900127ms iteration = 17 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
102082025-03-17T16:57:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
102092025-03-17T16:57:06.707ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
102102025-03-17T16:57:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
102112025-03-17T16:57:06.707ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
102122025-03-17T16:57:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
102132025-03-17T16:57:06.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
102142025-03-17T16:57:06.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
102152025-03-17T16:57:06.708ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
102162025-03-17T16:57:06.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
102172025-03-17T16:57:06.710ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102182025-03-17T16:57:06.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 8007234b-9b03-4ff0-af97-c65524f74b0a, lookup_type: ById(8007234b-9b03-4ff0-af97-c65524f74b0a) } result = Ok(())
102192025-03-17T16:57:06.713ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102202025-03-17T16:57:06.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102212025-03-17T16:57:06.715ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = e748635a-f614-4fd9-9b7f-4d3a8ef9a7f7 dns_zone_name = oxide.test found_so_far = 1 version = 2
102222025-03-17T16:57:06.715ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
102232025-03-17T16:57:06.715ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-17 16:41:51.295991 UTC dns_group = external generation = 2
102242025-03-17T16:57:06.715ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 122.756359ms iteration = 17 status = {"generation":2}
102252025-03-17T16:57:06.716ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 123.185038ms iteration = 33 status = 0
102262025-03-17T16:57:06.725ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
102272025-03-17T16:57:06.725ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 132.074919ms iteration = 33 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
102282025-03-17T16:57:06.740ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
102292025-03-17T16:57:06.741ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
102302025-03-17T16:57:06.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
102312025-03-17T16:57:06.741ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 147.838656ms iteration = 33 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
102322025-03-17T16:57:06.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
102332025-03-17T16:57:06.741ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
102342025-03-17T16:57:06.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
102352025-03-17T16:57:06.742ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102362025-03-17T16:57:06.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102372025-03-17T16:57:06.779ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
102382025-03-17T16:57:06.779ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102392025-03-17T16:57:06.780ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "586b0de2-f7a4-4a5a-bc39-0a21b79f3214", "content-length": "26", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
102402025-03-17T16:57:06.780ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102412025-03-17T16:57:06.782ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102422025-03-17T16:57:06.784ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
102432025-03-17T16:57:06.784ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 191.627214ms iteration = 17 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
102442025-03-17T16:57:06.804ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 211.233044ms iteration = 35 status = {"physical_disks_added":0}
102452025-03-17T16:57:06.829ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
102462025-03-17T16:57:06.877ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
102472025-03-17T16:57:06.900ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
102482025-03-17T16:57:06.900ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
102492025-03-17T16:57:06.900ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ec20f1-ab43-4543-abc8-3bafcd3f59ab", "content-length": "372", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
102502025-03-17T16:57:06.900ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102512025-03-17T16:57:06.901ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102522025-03-17T16:57:06.902ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102532025-03-17T16:57:06.902ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102542025-03-17T16:57:06.905ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
102552025-03-17T16:57:06.905ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
102562025-03-17T16:57:06.905ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
102572025-03-17T16:57:06.905ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
102582025-03-17T16:57:06.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
102592025-03-17T16:57:06.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
102602025-03-17T16:57:06.907ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
102612025-03-17T16:57:06.907ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102622025-03-17T16:57:06.907ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "303a31ad-3c61-42aa-a048-9032e9012732", "content-length": "26", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
102632025-03-17T16:57:06.907ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102642025-03-17T16:57:06.908ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102652025-03-17T16:57:06.912ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = a432a2ff-d40d-4500-9ffe-ee8ed6293383 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
102662025-03-17T16:57:06.912ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
102672025-03-17T16:57:06.912ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-17 16:41:50.383267 UTC dns_group = internal generation = 1
102682025-03-17T16:57:06.912ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 319.990083ms iteration = 17 status = {"generation":1}
102692025-03-17T16:57:06.942ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-17T16:41:04.847002Z, time_modified: 2025-03-17T16:41:04.847002Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 0218ebe8-928c-437a-97be-65be79de8ba1, db_silo: Silo { identity: SiloIdentity { id: 0218ebe8-928c-437a-97be-65be79de8ba1, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-17T16:41:51.234338Z, time_modified: 2025-03-17T16:41:51.234338Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3" }] }) }
102702025-03-17T16:57:06.942ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 349.829253ms iteration = 17 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]}},"default_endpoint":{"silo_id":"0218ebe8-928c-437a-97be-65be79de8ba1","tls_certs":["bc1f9ea2cf43682c65596d0be19c6325e5d7aca36740f1d32588e1bd533af9e3"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
102712025-03-17T16:57:06.945ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
102722025-03-17T16:57:06.945ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
102732025-03-17T16:57:06.945ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "632c76cf-08db-41c0-aafe-dfc242f6b555", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
102742025-03-17T16:57:06.945ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102752025-03-17T16:57:06.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102762025-03-17T16:57:06.948ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102772025-03-17T16:57:06.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102782025-03-17T16:57:06.951ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
102792025-03-17T16:57:06.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
102802025-03-17T16:57:06.955ZINFOnexus (BackgroundTasks): VPC route manager sled 8007234b-9b03-4ff0-af97-c65524f74b0a push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
102812025-03-17T16:57:06.955ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
102822025-03-17T16:57:06.955ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "93bfaed7-1b6e-45d0-9391-980a944f6862", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
102832025-03-17T16:57:06.957ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 364.506437ms iteration = 33 status = {}
102842025-03-17T16:57:06.972ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 380.234174ms iteration = 33 status = {}
102852025-03-17T16:57:06.987ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 zone_address = fd00:1122:3344:101::2
102862025-03-17T16:57:06.987ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102872025-03-17T16:57:06.987ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b35fa35-1165-4c62-b535-47e8aad08900", "content-length": "26", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
102882025-03-17T16:57:06.987ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102892025-03-17T16:57:06.989ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
102902025-03-17T16:57:06.996ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102912025-03-17T16:57:06.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102922025-03-17T16:57:06.998ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102932025-03-17T16:57:07.000ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
102942025-03-17T16:57:07.000ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102952025-03-17T16:57:07.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
102962025-03-17T16:57:07.002ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102972025-03-17T16:57:07.004ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
102982025-03-17T16:57:07.004ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
102992025-03-17T16:57:07.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103002025-03-17T16:57:07.043ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
103012025-03-17T16:57:07.043ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "881d28a6-3b58-40d4-b57d-9e59dfa169ee", "content-length": "2", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
103022025-03-17T16:57:07.045ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
103032025-03-17T16:57:07.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
103042025-03-17T16:57:07.048ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 455.943725ms iteration = 33 status = null
103052025-03-17T16:57:07.052ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
103062025-03-17T16:57:07.054ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
103072025-03-17T16:57:07.054ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "244b93e4-a27c-4cfe-8231-8299c029bb2f", "date": "Mon, 17 Mar 2025 16:57:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
103082025-03-17T16:57:07.054ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
103092025-03-17T16:57:07.055ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "d7405be2-e65e-402f-ab38-0912f062218d", "date": "Mon, 17 Mar 2025 16:57:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
103102025-03-17T16:57:07.055ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
103112025-03-17T16:57:07.057ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
103122025-03-17T16:57:07.057ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 465.585612ms iteration = 19 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
103132025-03-17T16:57:07.091ZDEBGnexus (BackgroundTasks): client request SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
103142025-03-17T16:57:07.092ZDEBGnexus (BackgroundTasks): client response SledAgent = 8007234b-9b03-4ff0-af97-c65524f74b0a background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2693337-fb36-4e20-b04f-8d05b89cf597", "content-length": "691", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
103152025-03-17T16:57:07.092ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
103162025-03-17T16:57:07.092ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
103172025-03-17T16:57:07.092ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
103182025-03-17T16:57:07.094ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 501.249816ms iteration = 33 status = {}
103192025-03-17T16:57:07.098ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
103202025-03-17T16:57:07.099ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "049edad4-628f-4ddb-af59-09703a6c790f", "date": "Mon, 17 Mar 2025 16:57:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103212025-03-17T16:57:07.099ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103222025-03-17T16:57:07.100ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b2e724b0-8f90-4e97-ac9f-2951298f2b1b", "date": "Mon, 17 Mar 2025 16:57:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103232025-03-17T16:57:07.100ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103242025-03-17T16:57:07.101ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "6e0f1272-06db-4e98-bf62-693b1cd06501", "date": "Mon, 17 Mar 2025 16:57:07 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103252025-03-17T16:57:07.101ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103262025-03-17T16:57:07.102ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103272025-03-17T16:57:07.102ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 509.89271ms iteration = 19 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
103282025-03-17T16:57:07.144ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103292025-03-17T16:57:07.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103302025-03-17T16:57:07.146ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 773a4daa-5a74-4111-a2ea-581acc25b78b, rack_id: 155b42a3-a710-4ba6-b3d8-718d3b712251, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(2e10b10e-db4f-43de-9c5b-f9e8d65f06ae) } rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
103312025-03-17T16:57:07.146ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103322025-03-17T16:57:07.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103332025-03-17T16:57:07.157ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
103342025-03-17T16:57:07.157ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "531258b8-9f35-4f13-94a7-8f0fdc8eeb59", "content-length": "134", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
103352025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
103362025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 2e10b10e-db4f-43de-9c5b-f9e8d65f06ae
103372025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
103382025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
103392025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103402025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103412025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103422025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
103432025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103442025-03-17T16:57:07.157ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
103452025-03-17T16:57:07.158ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251
103462025-03-17T16:57:07.158ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103472025-03-17T16:57:07.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103482025-03-17T16:57:07.160ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
103492025-03-17T16:57:07.160ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "07fece5b-c4ba-4f71-b70c-972dfacf4120", "content-length": "2", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
103502025-03-17T16:57:07.160ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
103512025-03-17T16:57:07.160ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "a787c7c0-6b03-458e-bdb5-13681153eded", "content-length": "52", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
103522025-03-17T16:57:07.160ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch_location = Switch0
103532025-03-17T16:57:07.160ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
103542025-03-17T16:57:07.254ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "09ae0f71-2317-4e79-9f29-c5a6626a15b5", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
103552025-03-17T16:57:07.254ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 switch = Switch0
103562025-03-17T16:57:07.254ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
103572025-03-17T16:57:07.255ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 155b42a3-a710-4ba6-b3d8-718d3b712251 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6753131-8ac7-4dda-9e04-dbc790713024", "content-length": "682", "date": "Mon, 17 Mar 2025 16:57:06 GMT"} })
103582025-03-17T16:57:07.255ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103592025-03-17T16:57:07.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103602025-03-17T16:57:07.259ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103612025-03-17T16:57:07.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103622025-03-17T16:57:07.262ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103632025-03-17T16:57:07.262ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103642025-03-17T16:57:07.264ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103652025-03-17T16:57:07.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103662025-03-17T16:57:07.271ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 677.912598ms iteration = 33 status = {}
103672025-03-17T16:57:07.837ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
103682025-03-17T16:57:07.837ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::b]:12221
103692025-03-17T16:57:07.867ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
103702025-03-17T16:57:07.867ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:54600
103712025-03-17T16:57:07.867ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:54600 req_id = a4fa7692-8994-438b-98a5-9938abd80965 roles = RoleSet { roles: {} } uri = /metrics/producers
103722025-03-17T16:57:07.868ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:54600 req_id = a4fa7692-8994-438b-98a5-9938abd80965 resource = Database result = Ok(()) uri = /metrics/producers
103732025-03-17T16:57:07.907ZINFOnexus: assigned collector to new producer collector_id = 1374f180-03c9-4b69-abf3-a0c92428bb9a file = nexus/src/app/oximeter.rs:76 producer_id = 807f84c2-ce3d-4190-8b1c-85a3b322601f
103742025-03-17T16:57:07.910ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 39308 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:54600 req_id = a4fa7692-8994-438b-98a5-9938abd80965 response_code = 201 uri = /metrics/producers
103752025-03-17T16:57:07.910ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "a4fa7692-8994-438b-98a5-9938abd80965", "content-length": "41", "date": "Mon, 17 Mar 2025 16:57:07 GMT"} })
103762025-03-17T16:57:07.910ZDEBGnexus (producer-registration-task): registered with nexus successfully
103772025-03-17T16:57:07.910ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
103782025-03-17T16:57:10.134ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 278 local_addr = [fd00:1122:3344:101::b]:48412 method = GET remote_addr = [fd00:1122:3344:101::d]:57156 req_id = ea4b7c15-97e5-4a9f-b067-e343b58af022 response_code = 200 uri = /807f84c2-ce3d-4190-8b1c-85a3b322601f