Max level shown:
12025-03-08T00:45:02.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 463 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 02710a5c-570f-4d5f-8d17-62d345107555 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
22025-03-08T00:45:10.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
32025-03-08T00:45:10.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42025-03-08T00:45:10.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52025-03-08T00:45:10.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
62025-03-08T00:45:10.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
72025-03-08T00:45:10.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82025-03-08T00:45:10.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92025-03-08T00:45:10.296ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
102025-03-08T00:45:10.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
112025-03-08T00:45:10.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
122025-03-08T00:45:10.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
132025-03-08T00:45:10.298ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.012519ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
142025-03-08T00:45:10.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
152025-03-08T00:45:10.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
162025-03-08T00:45:10.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
172025-03-08T00:45:10.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
182025-03-08T00:45:10.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
192025-03-08T00:45:10.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
202025-03-08T00:45:10.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
212025-03-08T00:45:10.319ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
222025-03-08T00:45:10.319ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.819387ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
232025-03-08T00:45:12.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 334 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = aa5d3a71-9523-4a16-9ef4-85297dbf66d8 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
242025-03-08T00:45:20.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
252025-03-08T00:45:20.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
262025-03-08T00:45:20.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
272025-03-08T00:45:20.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
282025-03-08T00:45:20.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
292025-03-08T00:45:20.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
302025-03-08T00:45:20.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
312025-03-08T00:45:20.296ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
322025-03-08T00:45:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
332025-03-08T00:45:20.298ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.180943ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
342025-03-08T00:45:20.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
352025-03-08T00:45:20.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
362025-03-08T00:45:20.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
372025-03-08T00:45:20.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
382025-03-08T00:45:20.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
392025-03-08T00:45:20.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
402025-03-08T00:45:20.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
412025-03-08T00:45:20.305ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
422025-03-08T00:45:20.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
432025-03-08T00:45:20.317ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
442025-03-08T00:45:20.317ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.193122ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
452025-03-08T00:45:22.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 385 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 1226dca9-b75f-4b28-82af-cf68470d0b22 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
462025-03-08T00:45:27.641ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = bc5796e4-e55d-4266-83a2-ed0ef4aa5cc5 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
472025-03-08T00:45:27.641ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = bc5796e4-e55d-4266-83a2-ed0ef4aa5cc5 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
482025-03-08T00:45:27.644ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2846 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = bc5796e4-e55d-4266-83a2-ed0ef4aa5cc5 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
492025-03-08T00:45:27.644ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 6376dab7-5044-4eef-a36e-4189a5ad10a1 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
502025-03-08T00:45:27.644ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 6376dab7-5044-4eef-a36e-4189a5ad10a1 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
512025-03-08T00:45:27.646ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2401 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 6376dab7-5044-4eef-a36e-4189a5ad10a1 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
522025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
532025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
542025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
552025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
562025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
572025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
582025-03-08T00:45:30.293ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-08 00:35:30.293311352 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
592025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
602025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
612025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
622025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
632025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
642025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
652025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
662025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
672025-03-08T00:45:30.293ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = a346f089-8033-4b92-b854-361d1da5cab2
682025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
692025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.68\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"a346f089-8033-4b92-b854-361d1da5cab2"}
702025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
712025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
722025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
732025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
742025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
752025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
762025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
772025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
782025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
792025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
802025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
812025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
822025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
832025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
842025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
852025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
862025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
872025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
882025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
892025-03-08T00:45:30.293ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
902025-03-08T00:45:30.294ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
912025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
922025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
932025-03-08T00:45:30.294ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("59287b7e-0d61-4586-b7c5-e79476927737.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1213, tv_nsec: 316089886 } })
942025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
952025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
962025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
972025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
982025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
992025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
1002025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
1012025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1022025-03-08T00:45:30.294ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
1032025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
1042025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
1052025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1062025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
1072025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1082025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
1092025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
1102025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1112025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
1122025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
1132025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
1142025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1152025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
1162025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1172025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
1182025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
1192025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1202025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1212025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
1222025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
1232025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
1242025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1252025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
1262025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
1272025-03-08T00:45:30.294ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
1282025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1292025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1302025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
1312025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
1322025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
1332025-03-08T00:45:30.294ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1ac98a83-1243-46f5-99db-fd29222026d8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("508de01a-93ec-4309-a59e-0cfd22b5027a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1213, tv_nsec: 316309741 } })
1342025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
1352025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1362025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1372025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
1382025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
1392025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
1402025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
1412025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.100578ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
1422025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1432025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1442025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1452025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
1462025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1472025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1482025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1492025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
1502025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1512025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1522025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1532025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1542025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1552025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1562025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
1572025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
1582025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
1592025-03-08T00:45:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.388145ms iteration = 6 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
1602025-03-08T00:45:30.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1612025-03-08T00:45:30.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1622025-03-08T00:45:30.296ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-08 00:35:30.293311352 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
1632025-03-08T00:45:30.296ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.924742ms iteration = 6 status = {"expiration":"2025-03-08T00:35:30.293311352Z","pruned":[]}
1642025-03-08T00:45:30.296ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1213, tv_nsec: 318423173 } })
1652025-03-08T00:45:30.296ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1662025-03-08T00:45:30.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
1672025-03-08T00:45:30.296ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1682025-03-08T00:45:30.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1692025-03-08T00:45:30.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1702025-03-08T00:45:30.297ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.366797ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
1712025-03-08T00:45:30.298ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1213, tv_nsec: 320366621 } })
1722025-03-08T00:45:30.298ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
1732025-03-08T00:45:30.299ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
1742025-03-08T00:45:30.299ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1752025-03-08T00:45:30.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1762025-03-08T00:45:30.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1772025-03-08T00:45:30.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1782025-03-08T00:45:30.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1792025-03-08T00:45:30.302ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1802025-03-08T00:45:30.302ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1812025-03-08T00:45:30.304ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
1822025-03-08T00:45:30.304ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.084328ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
1832025-03-08T00:45:30.304ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 10.394997ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
1842025-03-08T00:45:30.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
1852025-03-08T00:45:30.305ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
1862025-03-08T00:45:30.305ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
1872025-03-08T00:45:30.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
1882025-03-08T00:45:30.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
1892025-03-08T00:45:30.306ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
1902025-03-08T00:45:30.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
1912025-03-08T00:45:30.308ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1922025-03-08T00:45:30.310ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1932025-03-08T00:45:30.312ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1942025-03-08T00:45:30.312ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1952025-03-08T00:45:30.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1962025-03-08T00:45:30.313ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
1972025-03-08T00:45:30.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1982025-03-08T00:45:30.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1992025-03-08T00:45:30.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
2002025-03-08T00:45:30.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2012025-03-08T00:45:30.314ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2022025-03-08T00:45:30.315ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2032025-03-08T00:45:30.318ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
2042025-03-08T00:45:30.318ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
2052025-03-08T00:45:30.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
2062025-03-08T00:45:30.318ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
2072025-03-08T00:45:30.318ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2082025-03-08T00:45:30.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2092025-03-08T00:45:30.319ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2102025-03-08T00:45:30.321ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2112025-03-08T00:45:30.322ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
2122025-03-08T00:45:30.322ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2132025-03-08T00:45:30.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2142025-03-08T00:45:30.323ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2152025-03-08T00:45:30.323ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2162025-03-08T00:45:30.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2172025-03-08T00:45:30.325ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 31.068726ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
2182025-03-08T00:45:30.325ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2192025-03-08T00:45:30.326ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2202025-03-08T00:45:30.327ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2212025-03-08T00:45:30.327ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
2222025-03-08T00:45:30.327ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2232025-03-08T00:45:30.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2242025-03-08T00:45:30.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
2252025-03-08T00:45:30.330ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
2262025-03-08T00:45:30.331ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
2272025-03-08T00:45:30.331ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
2282025-03-08T00:45:30.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
2292025-03-08T00:45:30.331ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
2302025-03-08T00:45:30.333ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 38.801697ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
2312025-03-08T00:45:30.333ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2322025-03-08T00:45:30.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2332025-03-08T00:45:30.335ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
2342025-03-08T00:45:30.335ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
2352025-03-08T00:45:30.335ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
2362025-03-08T00:45:30.335ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2372025-03-08T00:45:30.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2382025-03-08T00:45:30.335ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
2392025-03-08T00:45:30.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
2402025-03-08T00:45:30.336ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
2412025-03-08T00:45:30.336ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 42.164861ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
2422025-03-08T00:45:30.338ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2432025-03-08T00:45:30.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2442025-03-08T00:45:30.338ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2452025-03-08T00:45:30.338ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
2462025-03-08T00:45:30.338ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2472025-03-08T00:45:30.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2482025-03-08T00:45:30.339ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2492025-03-08T00:45:30.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2502025-03-08T00:45:30.341ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2512025-03-08T00:45:30.342ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2522025-03-08T00:45:30.342ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
2532025-03-08T00:45:30.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
2542025-03-08T00:45:30.345ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2552025-03-08T00:45:30.345ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2562025-03-08T00:45:30.345ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
2572025-03-08T00:45:30.345ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2582025-03-08T00:45:30.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2592025-03-08T00:45:30.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2602025-03-08T00:45:30.350ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2612025-03-08T00:45:30.353ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2622025-03-08T00:45:30.353ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2632025-03-08T00:45:30.353ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 59.715443ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2642025-03-08T00:45:30.356ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
2652025-03-08T00:45:30.356ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2662025-03-08T00:45:30.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2672025-03-08T00:45:30.357ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2682025-03-08T00:45:30.357ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2692025-03-08T00:45:30.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
270 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
271 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
2722025-03-08T00:45:30.360ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
2732025-03-08T00:45:30.360ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2742025-03-08T00:45:30.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2752025-03-08T00:45:30.370ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2762025-03-08T00:45:30.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2772025-03-08T00:45:30.374ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2782025-03-08T00:45:30.377ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
2792025-03-08T00:45:30.377ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
2802025-03-08T00:45:30.377ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 83.519901ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
2812025-03-08T00:45:30.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
2822025-03-08T00:45:30.378ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
2832025-03-08T00:45:30.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
2842025-03-08T00:45:30.380ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
2852025-03-08T00:45:30.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
2862025-03-08T00:45:30.388ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2872025-03-08T00:45:30.388ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
2882025-03-08T00:45:30.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2892025-03-08T00:45:30.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
2902025-03-08T00:45:30.391ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2912025-03-08T00:45:30.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2922025-03-08T00:45:30.393ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
2932025-03-08T00:45:30.393ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
2942025-03-08T00:45:30.393ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
2952025-03-08T00:45:30.393ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2962025-03-08T00:45:30.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2972025-03-08T00:45:30.395ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
2982025-03-08T00:45:30.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
2992025-03-08T00:45:30.402ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
3002025-03-08T00:45:30.402ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
3012025-03-08T00:45:30.402ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 108.105937ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
3022025-03-08T00:45:30.410ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3032025-03-08T00:45:30.414ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3042025-03-08T00:45:30.414ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3052025-03-08T00:45:30.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3062025-03-08T00:45:30.444ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
3072025-03-08T00:45:30.458ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
3082025-03-08T00:45:30.458ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3092025-03-08T00:45:30.459ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bd744da-c8a5-4dbd-a9f8-728e807e7072", "content-length": "26", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
3102025-03-08T00:45:30.459ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3112025-03-08T00:45:30.461ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3122025-03-08T00:45:30.490ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3132025-03-08T00:45:30.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3142025-03-08T00:45:30.502ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
3152025-03-08T00:45:30.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
3162025-03-08T00:45:30.504ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
3172025-03-08T00:45:30.505ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3182025-03-08T00:45:30.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
3192025-03-08T00:45:30.507ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3202025-03-08T00:45:30.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3212025-03-08T00:45:30.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
3222025-03-08T00:45:30.507ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3232025-03-08T00:45:30.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3242025-03-08T00:45:30.526ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
3252025-03-08T00:45:30.526ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 233.077841ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
3262025-03-08T00:45:30.547ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
3272025-03-08T00:45:30.547ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3282025-03-08T00:45:30.547ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c6a0c10-e15e-47e5-a77c-352b82452526", "content-length": "26", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
3292025-03-08T00:45:30.547ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3302025-03-08T00:45:30.549ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3312025-03-08T00:45:30.551ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
3322025-03-08T00:45:30.586ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
3332025-03-08T00:45:30.586ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 293.38247ms iteration = 11 status = {}
3342025-03-08T00:45:30.623ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
3352025-03-08T00:45:30.623ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3362025-03-08T00:45:30.624ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef50d83b-2890-48ec-b15a-0b9ea827493e", "content-length": "372", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
3372025-03-08T00:45:30.624ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3382025-03-08T00:45:30.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3392025-03-08T00:45:30.625ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 331.633192ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3402025-03-08T00:45:30.627ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1213, tv_nsec: 649468552 } })
3412025-03-08T00:45:30.627ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
3422025-03-08T00:45:30.628ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3432025-03-08T00:45:30.630ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3442025-03-08T00:45:30.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3452025-03-08T00:45:30.631ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
3462025-03-08T00:45:30.631ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3472025-03-08T00:45:30.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3482025-03-08T00:45:30.632ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
3492025-03-08T00:45:30.632ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 338.262056ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
3502025-03-08T00:45:30.637ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3512025-03-08T00:45:30.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
3522025-03-08T00:45:30.640ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
3532025-03-08T00:45:30.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
3542025-03-08T00:45:30.643ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
3552025-03-08T00:45:30.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
3562025-03-08T00:45:30.646ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
3572025-03-08T00:45:30.646ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3582025-03-08T00:45:30.647ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "31bf3218-bcc7-4bbe-bcb2-733bd6a85ff0", "content-length": "2", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
3592025-03-08T00:45:30.647ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
3602025-03-08T00:45:30.649ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 355.545073ms iteration = 2 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"87826585-0843-40d2-a3ce-382fa54dbc5c","time":"2025-03-08T00:40:30.730769703Z"},{"operation":"list","target_sled":"87826585-0843-40d2-a3ce-382fa54dbc5c","time":"2025-03-08T00:45:30.647352828Z"}]}
3612025-03-08T00:45:30.655ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3622025-03-08T00:45:30.655ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
3632025-03-08T00:45:30.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
3642025-03-08T00:45:30.663ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3652025-03-08T00:45:30.663ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fb23942-e23f-4aa6-820a-84baa7d84e62", "content-length": "1", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
3662025-03-08T00:45:30.665ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3672025-03-08T00:45:30.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3682025-03-08T00:45:30.674ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 380.925509ms iteration = 11 status = null
3692025-03-08T00:45:30.677ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3702025-03-08T00:45:30.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
3712025-03-08T00:45:30.684ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
3722025-03-08T00:45:30.684ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "8a9af4ba-a730-45ac-8f0f-0514c290ddb6", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3732025-03-08T00:45:30.684ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3742025-03-08T00:45:30.686ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 392.235919ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
3752025-03-08T00:45:30.686ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "970c8399-3d6d-418f-adab-66e579c060cd", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3762025-03-08T00:45:30.686ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3772025-03-08T00:45:30.689ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
3782025-03-08T00:45:30.689ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 395.995181ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
3792025-03-08T00:45:30.689ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
3802025-03-08T00:45:30.689ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 0 version = 2
3812025-03-08T00:45:30.689ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3822025-03-08T00:45:30.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3832025-03-08T00:45:30.690ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
3842025-03-08T00:45:30.690ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 396.574045ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
3852025-03-08T00:45:30.690ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 396.752579ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
3862025-03-08T00:45:30.691ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3872025-03-08T00:45:30.692ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3882025-03-08T00:45:30.695ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 401.420265ms iteration = 12 status = {"physical_disks_added":0}
3892025-03-08T00:45:30.695ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3902025-03-08T00:45:30.696ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
3912025-03-08T00:45:30.696ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
3922025-03-08T00:45:30.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
3932025-03-08T00:45:30.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3942025-03-08T00:45:30.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3952025-03-08T00:45:30.701ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3962025-03-08T00:45:30.701ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3972025-03-08T00:45:30.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3982025-03-08T00:45:30.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3992025-03-08T00:45:30.701ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
4002025-03-08T00:45:30.702ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "56b5843d-1a9c-4e37-8e45-154335af1ba6", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4012025-03-08T00:45:30.702ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4022025-03-08T00:45:30.704ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4032025-03-08T00:45:30.704ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4042025-03-08T00:45:30.705ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "31502b22-36e3-4f92-9eec-74d03f45e7ba", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4052025-03-08T00:45:30.705ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4062025-03-08T00:45:30.706ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4072025-03-08T00:45:30.707ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 1 version = 2
4082025-03-08T00:45:30.707ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
4092025-03-08T00:45:30.707ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-08 00:40:32.180226 UTC dns_group = external generation = 2
4102025-03-08T00:45:30.707ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 413.843032ms iteration = 6 status = {"generation":2}
4112025-03-08T00:45:30.707ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d2b38c32-a3f4-4723-93aa-1b15123ad875", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4122025-03-08T00:45:30.707ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4132025-03-08T00:45:30.708ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4142025-03-08T00:45:30.708ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4152025-03-08T00:45:30.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4162025-03-08T00:45:30.710ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
4172025-03-08T00:45:30.710ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4182025-03-08T00:45:30.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4192025-03-08T00:45:30.712ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4202025-03-08T00:45:30.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4212025-03-08T00:45:30.717ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
4222025-03-08T00:45:30.717ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4232025-03-08T00:45:30.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4242025-03-08T00:45:30.718ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4252025-03-08T00:45:30.718ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 425.470398ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
4262025-03-08T00:45:30.724ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
4272025-03-08T00:45:30.724ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
4282025-03-08T00:45:30.725ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "51214fc1-3b37-4d84-b963-4d111be27f2b", "content-length": "26", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
4292025-03-08T00:45:30.725ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
4302025-03-08T00:45:30.725ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 431.515328ms iteration = 11 status = 0
4312025-03-08T00:45:30.727ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
4322025-03-08T00:45:30.731ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4332025-03-08T00:45:30.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4342025-03-08T00:45:30.735ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
4352025-03-08T00:45:30.735ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
4362025-03-08T00:45:30.735ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d8b78f9f-7252-4995-9631-f18bba0c23ae", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
4372025-03-08T00:45:30.735ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4382025-03-08T00:45:30.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4392025-03-08T00:45:30.738ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 443.878572ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
4402025-03-08T00:45:30.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4412025-03-08T00:45:30.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4422025-03-08T00:45:30.742ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4432025-03-08T00:45:30.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4442025-03-08T00:45:30.746ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
4452025-03-08T00:45:30.746ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
4462025-03-08T00:45:30.747ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ba036b0e-ea5e-4c8b-82b1-4fa981879c12", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
4472025-03-08T00:45:30.749ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 455.145901ms iteration = 11 status = {}
4482025-03-08T00:45:30.763ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-08T00:40:26.976319Z, time_modified: 2025-03-08T00:40:26.976319Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }) }
4492025-03-08T00:45:30.763ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 469.846323ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]}},"default_endpoint":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
4502025-03-08T00:45:30.765ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
4512025-03-08T00:45:30.765ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
4522025-03-08T00:45:30.765ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4532025-03-08T00:45:30.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4542025-03-08T00:45:30.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4552025-03-08T00:45:30.769ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4562025-03-08T00:45:30.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4572025-03-08T00:45:30.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4582025-03-08T00:45:30.770ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4592025-03-08T00:45:30.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4602025-03-08T00:45:30.773ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
4612025-03-08T00:45:30.773ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
4622025-03-08T00:45:30.773ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-08 00:40:30.694594 UTC dns_group = internal generation = 1
4632025-03-08T00:45:30.773ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 479.663696ms iteration = 6 status = {"generation":1}
4642025-03-08T00:45:30.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4652025-03-08T00:45:30.781ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
4662025-03-08T00:45:30.781ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4672025-03-08T00:45:30.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4682025-03-08T00:45:30.784ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4692025-03-08T00:45:30.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4702025-03-08T00:45:30.785ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4712025-03-08T00:45:30.787ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
4722025-03-08T00:45:30.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4732025-03-08T00:45:30.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4742025-03-08T00:45:30.790ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4752025-03-08T00:45:30.792ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
4762025-03-08T00:45:30.792ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
4772025-03-08T00:45:30.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4782025-03-08T00:45:30.797ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4792025-03-08T00:45:30.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4802025-03-08T00:45:30.799ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4812025-03-08T00:45:30.800ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
4822025-03-08T00:45:30.800ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4832025-03-08T00:45:30.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4842025-03-08T00:45:30.805ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4852025-03-08T00:45:30.808ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4862025-03-08T00:45:30.808ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4872025-03-08T00:45:30.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4882025-03-08T00:45:30.808ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4892025-03-08T00:45:30.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4902025-03-08T00:45:30.810ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4912025-03-08T00:45:30.810ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4922025-03-08T00:45:30.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4932025-03-08T00:45:30.812ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
4942025-03-08T00:45:30.812ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4952025-03-08T00:45:30.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4962025-03-08T00:45:30.813ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4972025-03-08T00:45:30.813ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4982025-03-08T00:45:30.815ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4992025-03-08T00:45:30.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5002025-03-08T00:45:30.816ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5012025-03-08T00:45:30.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5022025-03-08T00:45:30.817ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5032025-03-08T00:45:30.817ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5042025-03-08T00:45:30.817ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5052025-03-08T00:45:30.818ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5062025-03-08T00:45:30.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5072025-03-08T00:45:30.819ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5082025-03-08T00:45:30.819ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5092025-03-08T00:45:30.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5102025-03-08T00:45:30.820ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5112025-03-08T00:45:30.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5122025-03-08T00:45:30.827ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5132025-03-08T00:45:30.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5142025-03-08T00:45:30.829ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5152025-03-08T00:45:30.832ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5162025-03-08T00:45:30.832ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5172025-03-08T00:45:30.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5182025-03-08T00:45:30.837ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5192025-03-08T00:45:30.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5202025-03-08T00:45:30.839ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5212025-03-08T00:45:30.841ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5222025-03-08T00:45:30.842ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5232025-03-08T00:45:30.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5242025-03-08T00:45:30.847ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5252025-03-08T00:45:30.847ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5262025-03-08T00:45:30.855ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5272025-03-08T00:45:30.855ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5282025-03-08T00:45:30.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5292025-03-08T00:45:30.858ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
5302025-03-08T00:45:30.859ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5312025-03-08T00:45:30.859ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5322025-03-08T00:45:30.859ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5332025-03-08T00:45:30.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5342025-03-08T00:45:30.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5352025-03-08T00:45:30.862ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5362025-03-08T00:45:30.862ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5372025-03-08T00:45:30.864ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5382025-03-08T00:45:30.864ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5392025-03-08T00:45:30.867ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5402025-03-08T00:45:30.867ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5412025-03-08T00:45:30.870ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5422025-03-08T00:45:30.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5432025-03-08T00:45:30.889ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5442025-03-08T00:45:30.892ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5452025-03-08T00:45:30.892ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5462025-03-08T00:45:30.892ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5472025-03-08T00:45:30.894ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5482025-03-08T00:45:30.894ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5492025-03-08T00:45:30.896ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5502025-03-08T00:45:30.898ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5512025-03-08T00:45:30.898ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5522025-03-08T00:45:30.898ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5532025-03-08T00:45:30.903ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5542025-03-08T00:45:30.903ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5552025-03-08T00:45:30.904ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5562025-03-08T00:45:30.906ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5572025-03-08T00:45:30.906ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5582025-03-08T00:45:30.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5592025-03-08T00:45:30.921ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5602025-03-08T00:45:30.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5612025-03-08T00:45:30.926ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
5622025-03-08T00:45:30.926ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c4caaf-5367-4c73-9da9-af8edefeb914", "content-length": "691", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
5632025-03-08T00:45:30.926ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
5642025-03-08T00:45:30.926ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
5652025-03-08T00:45:30.926ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
5662025-03-08T00:45:30.927ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 633.760218ms iteration = 11 status = {}
5672025-03-08T00:45:30.928ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5682025-03-08T00:45:30.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5692025-03-08T00:45:30.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5702025-03-08T00:45:30.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5712025-03-08T00:45:30.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5722025-03-08T00:45:30.933ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5732025-03-08T00:45:30.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5742025-03-08T00:45:30.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5752025-03-08T00:45:30.969ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
5762025-03-08T00:45:30.969ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6716acef-87a5-4af0-b1ff-918a7aeaecc6", "content-length": "134", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
5772025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
5782025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
5792025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
5802025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
5812025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
5822025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
5832025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
5842025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
5852025-03-08T00:45:30.969ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5862025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
5872025-03-08T00:45:30.969ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
5882025-03-08T00:45:30.970ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
5892025-03-08T00:45:30.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5902025-03-08T00:45:30.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5912025-03-08T00:45:30.971ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5922025-03-08T00:45:30.973ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
5932025-03-08T00:45:30.973ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e468c39-4a68-4cba-a081-3c81d999a8bd", "content-length": "2", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
5942025-03-08T00:45:30.973ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
5952025-03-08T00:45:30.973ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "9233f45b-a812-4f06-a860-9aa43a09d067", "content-length": "52", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
5962025-03-08T00:45:30.973ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
5972025-03-08T00:45:30.973ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
5982025-03-08T00:45:30.974ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
5992025-03-08T00:45:30.974ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
6002025-03-08T00:45:30.974ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
6012025-03-08T00:45:30.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
6022025-03-08T00:45:30.974ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6032025-03-08T00:45:30.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6042025-03-08T00:45:30.976ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6052025-03-08T00:45:30.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6062025-03-08T00:45:30.977ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
6072025-03-08T00:45:30.979ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 683.765061ms iteration = 3 status = {"blueprint_id":"a346f089-8033-4b92-b854-361d1da5cab2","inventory_collection_id":"5d660b6a-29ce-4cae-9812-4763cac396ab","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":2,"num_already_tombstoned":0,"num_inserted":7,"num_not_in_inventory":0,"num_tombstoned":0}}}
6082025-03-08T00:45:31.006ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6092025-03-08T00:45:31.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6102025-03-08T00:45:31.008ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6112025-03-08T00:45:31.013ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
6122025-03-08T00:45:31.013ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6132025-03-08T00:45:31.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6142025-03-08T00:45:31.026ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6152025-03-08T00:45:31.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6162025-03-08T00:45:31.029ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6172025-03-08T00:45:31.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6182025-03-08T00:45:31.032ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6192025-03-08T00:45:31.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6202025-03-08T00:45:31.034ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6212025-03-08T00:45:31.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6222025-03-08T00:45:31.037ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6232025-03-08T00:45:31.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6242025-03-08T00:45:31.070ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6252025-03-08T00:45:31.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6262025-03-08T00:45:31.072ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6272025-03-08T00:45:31.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6282025-03-08T00:45:31.103ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6292025-03-08T00:45:31.107ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
6302025-03-08T00:45:31.107ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6312025-03-08T00:45:31.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6322025-03-08T00:45:31.108ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6332025-03-08T00:45:31.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6342025-03-08T00:45:31.110ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6352025-03-08T00:45:31.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6362025-03-08T00:45:31.112ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6372025-03-08T00:45:31.115ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
6382025-03-08T00:45:31.115ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6392025-03-08T00:45:31.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6402025-03-08T00:45:31.128ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 0c3d6881-8d48-4d96-9ed0-85d27344279b, kind: Service { id: 4d603338-4154-45db-a26b-b6679dc7f0ec }, name: Name("nexus-4d603338-4154-45db-a26b-b6679dc7f0ec"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 106])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 3723c067-03a8-4ed5-9a97-5c3ee10bca5c, kind: Service { id: 841ee309-4e1e-443f-ae06-2ad5eb454a8d }, name: Name("nexus-841ee309-4e1e-443f-ae06-2ad5eb454a8d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 133])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d79ffed2-490d-44d6-943a-e4d940d7938a, kind: Service { id: 8b94fb8b-b81f-40e3-a7d8-ae559205ae7f }, name: Name("nexus-8b94fb8b-b81f-40e3-a7d8-ae559205ae7f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 217])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 4d0825be-9769-464b-9eb9-39c5eebe5d62, kind: Service { id: da0d8428-046e-431d-9a82-470b80ca232a }, name: Name("external-dns-da0d8428-046e-431d-9a82-470b80ca232a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 176])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d157e977-c32e-4a22-a13f-9a7473eb90e5, kind: Service { id: 59287b7e-0d61-4586-b7c5-e79476927737 }, name: Name("external-dns-59287b7e-0d61-4586-b7c5-e79476927737"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 101])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
6412025-03-08T00:45:31.128ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6422025-03-08T00:45:31.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6432025-03-08T00:45:31.130ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6442025-03-08T00:45:31.134ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
6452025-03-08T00:45:31.134ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6462025-03-08T00:45:31.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6472025-03-08T00:45:31.197ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "3bf925f1-c522-4521-910c-d227522f01b1", "date": "Sat, 08 Mar 2025 00:45:30 GMT"} })
6482025-03-08T00:45:31.197ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
6492025-03-08T00:45:31.197ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
6502025-03-08T00:45:31.198ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ccf3f76-e95a-4800-a9ef-32281e0616c8", "content-length": "682", "date": "Sat, 08 Mar 2025 00:45:31 GMT"} })
6512025-03-08T00:45:31.198ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6522025-03-08T00:45:31.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6532025-03-08T00:45:31.202ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6542025-03-08T00:45:31.202ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6552025-03-08T00:45:31.205ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6562025-03-08T00:45:31.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6572025-03-08T00:45:31.208ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6582025-03-08T00:45:31.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6592025-03-08T00:45:31.214ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 920.611556ms iteration = 11 status = {}
6602025-03-08T00:45:31.237ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
6612025-03-08T00:45:31.237ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4d0825be-9769-464b-9eb9-39c5eebe5d62, kind: Service { id: da0d8428-046e-431d-9a82-470b80ca232a }, name: Name("external-dns-da0d8428-046e-431d-9a82-470b80ca232a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 176])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d157e977-c32e-4a22-a13f-9a7473eb90e5, kind: Service { id: 59287b7e-0d61-4586-b7c5-e79476927737 }, name: Name("external-dns-59287b7e-0d61-4586-b7c5-e79476927737"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 101])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 0c3d6881-8d48-4d96-9ed0-85d27344279b, kind: Service { id: 4d603338-4154-45db-a26b-b6679dc7f0ec }, name: Name("nexus-4d603338-4154-45db-a26b-b6679dc7f0ec"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 106])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 3723c067-03a8-4ed5-9a97-5c3ee10bca5c, kind: Service { id: 841ee309-4e1e-443f-ae06-2ad5eb454a8d }, name: Name("nexus-841ee309-4e1e-443f-ae06-2ad5eb454a8d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 133])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d79ffed2-490d-44d6-943a-e4d940d7938a, kind: Service { id: 8b94fb8b-b81f-40e3-a7d8-ae559205ae7f }, name: Name("nexus-8b94fb8b-b81f-40e3-a7d8-ae559205ae7f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 217])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
6622025-03-08T00:45:31.237ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
6632025-03-08T00:45:31.357ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 87826585-0843-40d2-a3ce-382fa54dbc5c, time_created: 2025-03-08T00:40:28.174204Z, time_modified: 2025-03-08T00:40:28.174204Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
6642025-03-08T00:45:31.357ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
6652025-03-08T00:45:31.357ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6662025-03-08T00:45:31.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6672025-03-08T00:45:31.359ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6682025-03-08T00:45:31.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6692025-03-08T00:45:31.361ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6702025-03-08T00:45:31.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
6712025-03-08T00:45:31.399ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
6722025-03-08T00:45:31.399ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "d019e141-99cf-439e-be3a-99e4c42088f7", "date": "Sat, 08 Mar 2025 00:45:31 GMT"} })
6732025-03-08T00:45:31.401ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
6742025-03-08T00:45:31.401ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 1.107089226s
6752025-03-08T00:45:31.401ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 1.107096896s iteration = 2 status = {}
6762025-03-08T00:45:32.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 385 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 4532bbb7-e272-4e77-8a0b-b636ee7567b1 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
6772025-03-08T00:45:40.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
6782025-03-08T00:45:40.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6792025-03-08T00:45:40.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6802025-03-08T00:45:40.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
6812025-03-08T00:45:40.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6822025-03-08T00:45:40.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6832025-03-08T00:45:40.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6842025-03-08T00:45:40.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6852025-03-08T00:45:40.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6862025-03-08T00:45:40.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6872025-03-08T00:45:40.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6882025-03-08T00:45:40.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6892025-03-08T00:45:40.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6902025-03-08T00:45:40.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
6912025-03-08T00:45:40.305ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6922025-03-08T00:45:40.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6932025-03-08T00:45:40.326ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
6942025-03-08T00:45:40.326ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 33.779243ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
6952025-03-08T00:45:40.330ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6962025-03-08T00:45:40.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6972025-03-08T00:45:40.361ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.679265ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6982025-03-08T00:45:42.640ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 1ca9f6af-4284-497e-8213-daa246897571 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
6992025-03-08T00:45:42.641ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 1ca9f6af-4284-497e-8213-daa246897571 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
7002025-03-08T00:45:42.644ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3185 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 1ca9f6af-4284-497e-8213-daa246897571 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
7012025-03-08T00:45:42.644ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 95ca74eb-5360-4dbf-840f-82acfe670e6c roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
7022025-03-08T00:45:42.644ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 95ca74eb-5360-4dbf-840f-82acfe670e6c resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
7032025-03-08T00:45:42.646ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2199 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 95ca74eb-5360-4dbf-840f-82acfe670e6c response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
7042025-03-08T00:45:42.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 388 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = e9d52954-e00a-4a9b-bfb9-269dc549213f response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
7052025-03-08T00:45:50.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
7062025-03-08T00:45:50.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7072025-03-08T00:45:50.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7082025-03-08T00:45:50.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
7092025-03-08T00:45:50.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7102025-03-08T00:45:50.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7112025-03-08T00:45:50.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7122025-03-08T00:45:50.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7132025-03-08T00:45:50.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7142025-03-08T00:45:50.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7152025-03-08T00:45:50.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7162025-03-08T00:45:50.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7172025-03-08T00:45:50.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7182025-03-08T00:45:50.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
7192025-03-08T00:45:50.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7202025-03-08T00:45:50.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7212025-03-08T00:45:50.320ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
7222025-03-08T00:45:50.320ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.452538ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
7232025-03-08T00:45:50.329ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7242025-03-08T00:45:50.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7252025-03-08T00:45:50.360ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.142172ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7262025-03-08T00:45:52.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 371 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = c22476b7-6fae-47af-8648-3d78a559d61c response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
7272025-03-08T00:45:57.641ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 06849e32-9cea-4054-bda4-b6addc30aa94 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
7282025-03-08T00:45:57.641ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 06849e32-9cea-4054-bda4-b6addc30aa94 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
7292025-03-08T00:45:57.644ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3234 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 06849e32-9cea-4054-bda4-b6addc30aa94 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
7302025-03-08T00:45:57.644ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 24fc63f3-a9ed-4e5f-8059-f7d2ad5fff23 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
7312025-03-08T00:45:57.644ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 24fc63f3-a9ed-4e5f-8059-f7d2ad5fff23 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
7322025-03-08T00:45:57.646ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2452 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:48539 req_id = 24fc63f3-a9ed-4e5f-8059-f7d2ad5fff23 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
7332025-03-08T00:46:00.293ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
7342025-03-08T00:46:00.293ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
7352025-03-08T00:46:00.293ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7362025-03-08T00:46:00.293ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7372025-03-08T00:46:00.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
7382025-03-08T00:46:00.293ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
7392025-03-08T00:46:00.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7402025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7412025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7422025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7432025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
7442025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
7452025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7462025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7472025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
7482025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7492025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
7502025-03-08T00:46:00.294ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
7512025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
7522025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7532025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7542025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
7552025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
7562025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7572025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
7582025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
7592025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
7602025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7612025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
7622025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7632025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
7642025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7652025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
7662025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7672025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
7682025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7692025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
7702025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7712025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
7722025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
7732025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7742025-03-08T00:46:00.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7752025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7762025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7772025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7782025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7792025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7802025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7812025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7822025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7832025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
7842025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
7852025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7862025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7872025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7882025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7892025-03-08T00:46:00.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7902025-03-08T00:46:00.296ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1243, tv_nsec: 318622398 } })
7912025-03-08T00:46:00.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
7922025-03-08T00:46:00.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7932025-03-08T00:46:00.297ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1243, tv_nsec: 319518760 } })
7942025-03-08T00:46:00.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7952025-03-08T00:46:00.297ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
7962025-03-08T00:46:00.297ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.082131ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
7972025-03-08T00:46:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
7982025-03-08T00:46:00.300ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7992025-03-08T00:46:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8002025-03-08T00:46:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8012025-03-08T00:46:00.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8022025-03-08T00:46:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8032025-03-08T00:46:00.307ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8042025-03-08T00:46:00.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8052025-03-08T00:46:00.308ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1243, tv_nsec: 331240320 } })
8062025-03-08T00:46:00.309ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
8072025-03-08T00:46:00.360ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8082025-03-08T00:46:00.361ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 66.609313ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
8092025-03-08T00:46:00.362ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8102025-03-08T00:46:00.362ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8112025-03-08T00:46:00.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8122025-03-08T00:46:00.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8132025-03-08T00:46:00.362ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 68.030459ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
8142025-03-08T00:46:00.363ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8152025-03-08T00:46:00.363ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8162025-03-08T00:46:00.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8172025-03-08T00:46:00.364ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
8182025-03-08T00:46:00.364ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8192025-03-08T00:46:00.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8202025-03-08T00:46:00.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8212025-03-08T00:46:00.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8222025-03-08T00:46:00.366ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8232025-03-08T00:46:00.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8242025-03-08T00:46:00.366ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8252025-03-08T00:46:00.368ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8262025-03-08T00:46:00.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8272025-03-08T00:46:00.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
8282025-03-08T00:46:00.370ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8292025-03-08T00:46:00.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
8302025-03-08T00:46:00.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
8312025-03-08T00:46:00.370ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8322025-03-08T00:46:00.370ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8332025-03-08T00:46:00.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8342025-03-08T00:46:00.371ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 76.227641ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
8352025-03-08T00:46:00.372ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8362025-03-08T00:46:00.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8372025-03-08T00:46:00.373ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8382025-03-08T00:46:00.373ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
8392025-03-08T00:46:00.373ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8402025-03-08T00:46:00.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8412025-03-08T00:46:00.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
8422025-03-08T00:46:00.373ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8432025-03-08T00:46:00.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8442025-03-08T00:46:00.376ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8452025-03-08T00:46:00.376ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8462025-03-08T00:46:00.378ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
8472025-03-08T00:46:00.378ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8482025-03-08T00:46:00.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8492025-03-08T00:46:00.380ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8502025-03-08T00:46:00.380ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8512025-03-08T00:46:00.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8522025-03-08T00:46:00.382ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8532025-03-08T00:46:00.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8542025-03-08T00:46:00.383ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8552025-03-08T00:46:00.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8562025-03-08T00:46:00.384ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
8572025-03-08T00:46:00.384ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
8582025-03-08T00:46:00.384ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
8592025-03-08T00:46:00.384ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 89.499259ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
8602025-03-08T00:46:00.384ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e430e116-a9d4-4a91-ba7c-7c33c846efa0", "content-length": "26", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
8612025-03-08T00:46:00.384ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
8622025-03-08T00:46:00.385ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8632025-03-08T00:46:00.386ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
8642025-03-08T00:46:00.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8652025-03-08T00:46:00.388ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8662025-03-08T00:46:00.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8672025-03-08T00:46:00.390ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
8682025-03-08T00:46:00.390ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 97.228639ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
8692025-03-08T00:46:00.397ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
8702025-03-08T00:46:00.397ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
871 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
8722025-03-08T00:46:00.398ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "96b3dcbc-b7e2-4e9a-b0fb-b37606494cc0", "content-length": "26", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
8732025-03-08T00:46:00.398ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
8742025-03-08T00:46:00.400ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
8752025-03-08T00:46:00.409ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8762025-03-08T00:46:00.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8772025-03-08T00:46:00.413ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
8782025-03-08T00:46:00.413ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
8792025-03-08T00:46:00.414ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ac4ca4-9447-4892-8f6b-5129addc71b1", "content-length": "26", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
8802025-03-08T00:46:00.414ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
8812025-03-08T00:46:00.415ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
8822025-03-08T00:46:00.419ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 124.733767ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8832025-03-08T00:46:00.421ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8842025-03-08T00:46:00.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8852025-03-08T00:46:00.423ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8862025-03-08T00:46:00.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8872025-03-08T00:46:00.427ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8882025-03-08T00:46:00.427ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
8892025-03-08T00:46:00.427ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8902025-03-08T00:46:00.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8912025-03-08T00:46:00.428ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 133.813032ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
8922025-03-08T00:46:00.430ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8932025-03-08T00:46:00.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8942025-03-08T00:46:00.434ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8952025-03-08T00:46:00.438ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 143.640344ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
8962025-03-08T00:46:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8972025-03-08T00:46:00.439ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8982025-03-08T00:46:00.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8992025-03-08T00:46:00.443ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
9002025-03-08T00:46:00.443ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e26fe1-b108-464b-b620-d291a597078c", "content-length": "1", "date": "Sat, 08 Mar 2025 00:45:59 GMT"} })
9012025-03-08T00:46:00.444ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 149.628112ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
9022025-03-08T00:46:00.445ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9032025-03-08T00:46:00.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9042025-03-08T00:46:00.448ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 154.979724ms iteration = 12 status = null
9052025-03-08T00:46:00.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9062025-03-08T00:46:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
9072025-03-08T00:46:00.461ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9082025-03-08T00:46:00.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9092025-03-08T00:46:00.461ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 168.255451ms iteration = 12 status = {}
9102025-03-08T00:46:00.466ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 171.734748ms iteration = 12 status = 0
9112025-03-08T00:46:00.468ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9122025-03-08T00:46:00.468ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
9132025-03-08T00:46:00.468ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 173.479281ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
9142025-03-08T00:46:00.474ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
9152025-03-08T00:46:00.474ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 179.624702ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
9162025-03-08T00:46:00.499ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9172025-03-08T00:46:00.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9182025-03-08T00:46:00.503ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
9192025-03-08T00:46:00.503ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9202025-03-08T00:46:00.504ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b43b3d3-1990-4095-b626-56496bac67d2", "content-length": "372", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9212025-03-08T00:46:00.504ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9222025-03-08T00:46:00.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9232025-03-08T00:46:00.518ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
9242025-03-08T00:46:00.518ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
9252025-03-08T00:46:00.519ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "35ce819d-8499-4153-88ab-03b3d6b7dff9", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9262025-03-08T00:46:00.519ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9272025-03-08T00:46:00.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9282025-03-08T00:46:00.520ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9292025-03-08T00:46:00.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9302025-03-08T00:46:00.522ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 227.77656ms iteration = 13 status = {"physical_disks_added":0}
9312025-03-08T00:46:00.524ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9322025-03-08T00:46:00.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9332025-03-08T00:46:00.532ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
9342025-03-08T00:46:00.532ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9352025-03-08T00:46:00.532ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "1c570a8c-8804-4cca-ab5c-25a5142859b6", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9362025-03-08T00:46:00.533ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9372025-03-08T00:46:00.534ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 240.205677ms iteration = 12 status = {}
9382025-03-08T00:46:00.536ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9392025-03-08T00:46:00.536ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9402025-03-08T00:46:00.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9412025-03-08T00:46:00.537ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9422025-03-08T00:46:00.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9432025-03-08T00:46:00.537ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9442025-03-08T00:46:00.539ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
9452025-03-08T00:46:00.539ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9462025-03-08T00:46:00.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9472025-03-08T00:46:00.541ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9482025-03-08T00:46:00.541ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9492025-03-08T00:46:00.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9502025-03-08T00:46:00.554ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
9512025-03-08T00:46:00.555ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c45554a1-4842-405a-8130-075ecc1f84e7", "content-length": "134", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9522025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
9532025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
9542025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
9552025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
9562025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9572025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9582025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9592025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9602025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9612025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9622025-03-08T00:46:00.555ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
9632025-03-08T00:46:00.555ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9642025-03-08T00:46:00.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9652025-03-08T00:46:00.557ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
9662025-03-08T00:46:00.558ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "01190bba-8aec-4655-b848-e35f8e73eb0b", "content-length": "2", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9672025-03-08T00:46:00.558ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
9682025-03-08T00:46:00.558ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9b33676-095f-44a2-b4ec-7d71dbe10702", "content-length": "52", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9692025-03-08T00:46:00.558ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
9702025-03-08T00:46:00.558ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
9712025-03-08T00:46:00.646ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
9722025-03-08T00:46:00.685ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8880bfa6-415d-4ba5-8c6e-7f3346ee0450", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9732025-03-08T00:46:00.686ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
9742025-03-08T00:46:00.686ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
9752025-03-08T00:46:00.686ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "03a8d6b3-d71e-42bd-9903-1de90c5b5319", "content-length": "682", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9762025-03-08T00:46:00.686ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9772025-03-08T00:46:00.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9782025-03-08T00:46:00.687ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ddf72c0-7f57-4827-a9fc-51e1ad64e666", "content-length": "691", "date": "Sat, 08 Mar 2025 00:46:00 GMT"} })
9792025-03-08T00:46:00.687ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
9802025-03-08T00:46:00.687ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
9812025-03-08T00:46:00.687ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
9822025-03-08T00:46:00.690ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 395.486309ms iteration = 12 status = {}
9832025-03-08T00:46:00.700ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9842025-03-08T00:46:00.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9852025-03-08T00:46:00.704ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9862025-03-08T00:46:00.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9872025-03-08T00:46:00.707ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9882025-03-08T00:46:00.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9892025-03-08T00:46:00.716ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 422.146606ms iteration = 12 status = {}
9902025-03-08T00:46:02.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 7ff71eb0-217d-4d01-8845-a4eb81bfbeda response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
9912025-03-08T00:46:10.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
9922025-03-08T00:46:10.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9932025-03-08T00:46:10.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9942025-03-08T00:46:10.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
9952025-03-08T00:46:10.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9962025-03-08T00:46:10.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9972025-03-08T00:46:10.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9982025-03-08T00:46:10.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9992025-03-08T00:46:10.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10002025-03-08T00:46:10.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10012025-03-08T00:46:10.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10022025-03-08T00:46:10.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10032025-03-08T00:46:10.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10042025-03-08T00:46:10.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
10052025-03-08T00:46:10.305ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10062025-03-08T00:46:10.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10072025-03-08T00:46:10.314ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
10082025-03-08T00:46:10.314ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.414229ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
10092025-03-08T00:46:10.323ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10102025-03-08T00:46:10.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10112025-03-08T00:46:10.358ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.403389ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10122025-03-08T00:46:12.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 361 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = ff5d05ec-24ca-4fdf-ac40-db8244772d81 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
10132025-03-08T00:46:20.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
10142025-03-08T00:46:20.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10152025-03-08T00:46:20.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10162025-03-08T00:46:20.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10172025-03-08T00:46:20.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
10182025-03-08T00:46:20.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10192025-03-08T00:46:20.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10202025-03-08T00:46:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10212025-03-08T00:46:20.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10222025-03-08T00:46:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10232025-03-08T00:46:20.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10242025-03-08T00:46:20.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10252025-03-08T00:46:20.324ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10262025-03-08T00:46:20.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10272025-03-08T00:46:20.326ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.984879ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10282025-03-08T00:46:20.329ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10292025-03-08T00:46:20.332ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
10302025-03-08T00:46:20.332ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10312025-03-08T00:46:20.332ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10322025-03-08T00:46:20.344ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
10332025-03-08T00:46:20.344ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.633614ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
10342025-03-08T00:46:22.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 360 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 56ed763e-56f8-46a5-986b-50c825e9b9f9 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
10352025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
10362025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
10372025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10382025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10392025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
10402025-03-08T00:46:30.292ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-08 00:36:30.292882010 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
10412025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
10422025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
10432025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
10442025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
10452025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10462025-03-08T00:46:30.292ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10472025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
10482025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
10492025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
10502025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
10512025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10522025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
10532025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
10542025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
10552025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
10562025-03-08T00:46:30.293ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = a346f089-8033-4b92-b854-361d1da5cab2
10572025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.2\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"a346f089-8033-4b92-b854-361d1da5cab2"}
10582025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
10592025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10602025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10612025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
10622025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10632025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
10642025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10652025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10662025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10672025-03-08T00:46:30.293ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("59287b7e-0d61-4586-b7c5-e79476927737.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1273, tv_nsec: 315950702 } })
10682025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
10692025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
10702025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
10712025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
10722025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10732025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10742025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
10752025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
10762025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
10772025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
10782025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
10792025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
10802025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
10812025-03-08T00:46:30.293ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1ac98a83-1243-46f5-99db-fd29222026d8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("508de01a-93ec-4309-a59e-0cfd22b5027a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1273, tv_nsec: 316091876 } })
10822025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
10832025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
10842025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
10852025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
10862025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
10872025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
10882025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
10892025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
10902025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10912025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
10922025-03-08T00:46:30.293ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
10932025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10942025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
10952025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
10962025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10972025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
10982025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
10992025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
11002025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11012025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11022025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11032025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
11042025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
11052025-03-08T00:46:30.293ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
11062025-03-08T00:46:30.293ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11072025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
11082025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
11092025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11102025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
11112025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11122025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
11132025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11142025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11152025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11162025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11172025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11182025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
11192025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
11202025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11212025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11222025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
11232025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.23167ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
11242025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
11252025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.258171ms iteration = 7 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
11262025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
11272025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11282025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11292025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11302025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11312025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11322025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
11332025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11342025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11352025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11362025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
11372025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11382025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11392025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11402025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11412025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11422025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11432025-03-08T00:46:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11442025-03-08T00:46:30.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11452025-03-08T00:46:30.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11462025-03-08T00:46:30.295ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11472025-03-08T00:46:30.295ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.83087ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
11482025-03-08T00:46:30.296ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1273, tv_nsec: 319012618 } })
11492025-03-08T00:46:30.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
11502025-03-08T00:46:30.297ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
11512025-03-08T00:46:30.297ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11522025-03-08T00:46:30.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11532025-03-08T00:46:30.297ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-08 00:36:30.292882010 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
11542025-03-08T00:46:30.297ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.645405ms iteration = 7 status = {"expiration":"2025-03-08T00:36:30.292882010Z","pruned":[]}
11552025-03-08T00:46:30.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11562025-03-08T00:46:30.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11572025-03-08T00:46:30.299ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1273, tv_nsec: 322070003 } })
11582025-03-08T00:46:30.299ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
11592025-03-08T00:46:30.300ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11602025-03-08T00:46:30.301ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11612025-03-08T00:46:30.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11622025-03-08T00:46:30.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
11632025-03-08T00:46:30.304ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11642025-03-08T00:46:30.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11652025-03-08T00:46:30.307ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 13.290768ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
11662025-03-08T00:46:30.308ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11672025-03-08T00:46:30.311ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
11682025-03-08T00:46:30.311ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11692025-03-08T00:46:30.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11702025-03-08T00:46:30.313ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11712025-03-08T00:46:30.315ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11722025-03-08T00:46:30.315ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11732025-03-08T00:46:30.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11742025-03-08T00:46:30.317ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11752025-03-08T00:46:30.321ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
11762025-03-08T00:46:30.321ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11772025-03-08T00:46:30.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11782025-03-08T00:46:30.326ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11792025-03-08T00:46:30.328ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
11802025-03-08T00:46:30.328ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11812025-03-08T00:46:30.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11822025-03-08T00:46:30.330ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11832025-03-08T00:46:30.334ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
11842025-03-08T00:46:30.334ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11852025-03-08T00:46:30.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11862025-03-08T00:46:30.341ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11872025-03-08T00:46:30.347ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
11882025-03-08T00:46:30.350ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11892025-03-08T00:46:30.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11902025-03-08T00:46:30.353ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11912025-03-08T00:46:30.358ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11922025-03-08T00:46:30.358ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.650746ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
11932025-03-08T00:46:30.358ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11942025-03-08T00:46:30.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11952025-03-08T00:46:30.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11962025-03-08T00:46:30.359ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11972025-03-08T00:46:30.362ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 68.607073ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
11982025-03-08T00:46:30.364ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.369906ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11992025-03-08T00:46:30.364ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
12002025-03-08T00:46:30.364ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 70.693285ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
12012025-03-08T00:46:30.367ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12022025-03-08T00:46:30.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12032025-03-08T00:46:30.369ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12042025-03-08T00:46:30.369ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12052025-03-08T00:46:30.371ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
12062025-03-08T00:46:30.371ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
12072025-03-08T00:46:30.371ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 77.786359ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
12082025-03-08T00:46:30.373ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12092025-03-08T00:46:30.373ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12102025-03-08T00:46:30.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
12112025-03-08T00:46:30.373ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
12122025-03-08T00:46:30.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12132025-03-08T00:46:30.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
12142025-03-08T00:46:30.375ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 81.45684ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
12152025-03-08T00:46:30.375ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12162025-03-08T00:46:30.375ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12172025-03-08T00:46:30.375ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
12182025-03-08T00:46:30.375ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12192025-03-08T00:46:30.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12202025-03-08T00:46:30.376ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12212025-03-08T00:46:30.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12222025-03-08T00:46:30.379ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12232025-03-08T00:46:30.379ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12242025-03-08T00:46:30.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12252025-03-08T00:46:30.380ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12262025-03-08T00:46:30.380ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12272025-03-08T00:46:30.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12282025-03-08T00:46:30.381ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12292025-03-08T00:46:30.381ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12302025-03-08T00:46:30.381ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12312025-03-08T00:46:30.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
12322025-03-08T00:46:30.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12332025-03-08T00:46:30.382ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1273, tv_nsec: 404965558 } })
12342025-03-08T00:46:30.382ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
12352025-03-08T00:46:30.385ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12362025-03-08T00:46:30.385ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12372025-03-08T00:46:30.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12382025-03-08T00:46:30.385ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12392025-03-08T00:46:30.385ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12402025-03-08T00:46:30.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12412025-03-08T00:46:30.388ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12422025-03-08T00:46:30.390ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12432025-03-08T00:46:30.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12442025-03-08T00:46:30.390ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12452025-03-08T00:46:30.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12462025-03-08T00:46:30.392ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12472025-03-08T00:46:30.392ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12482025-03-08T00:46:30.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12492025-03-08T00:46:30.394ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12502025-03-08T00:46:30.394ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12512025-03-08T00:46:30.396ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 102.212832ms iteration = 4 status = {"nsuccess":5}
12522025-03-08T00:46:30.398ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
12532025-03-08T00:46:30.398ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12542025-03-08T00:46:30.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12552025-03-08T00:46:30.398ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12562025-03-08T00:46:30.398ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12572025-03-08T00:46:30.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1258 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
12592025-03-08T00:46:30.407ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
12602025-03-08T00:46:30.407ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 113.836529ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
12612025-03-08T00:46:30.413ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
12622025-03-08T00:46:30.413ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
12632025-03-08T00:46:30.413ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
12642025-03-08T00:46:30.413ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12652025-03-08T00:46:30.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12662025-03-08T00:46:30.419ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12672025-03-08T00:46:30.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
12682025-03-08T00:46:30.426ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12692025-03-08T00:46:30.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12702025-03-08T00:46:30.429ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12712025-03-08T00:46:30.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12722025-03-08T00:46:30.431ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12732025-03-08T00:46:30.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12742025-03-08T00:46:30.432ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12752025-03-08T00:46:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12762025-03-08T00:46:30.436ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12772025-03-08T00:46:30.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12782025-03-08T00:46:30.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12792025-03-08T00:46:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12802025-03-08T00:46:30.439ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
12812025-03-08T00:46:30.439ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
12822025-03-08T00:46:30.439ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
12832025-03-08T00:46:30.439ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
12842025-03-08T00:46:30.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12852025-03-08T00:46:30.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12862025-03-08T00:46:30.440ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12872025-03-08T00:46:30.442ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12882025-03-08T00:46:30.443ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12892025-03-08T00:46:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12902025-03-08T00:46:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
12912025-03-08T00:46:30.445ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12922025-03-08T00:46:30.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12932025-03-08T00:46:30.445ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 152.135034ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
12942025-03-08T00:46:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
12952025-03-08T00:46:30.446ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12962025-03-08T00:46:30.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12972025-03-08T00:46:30.448ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
12982025-03-08T00:46:30.448ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
12992025-03-08T00:46:30.448ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 154.889872ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
13002025-03-08T00:46:30.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13012025-03-08T00:46:30.453ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 159.887785ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13022025-03-08T00:46:30.453ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13032025-03-08T00:46:30.453ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
13042025-03-08T00:46:30.453ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13052025-03-08T00:46:30.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13062025-03-08T00:46:30.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
13072025-03-08T00:46:30.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13082025-03-08T00:46:30.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13092025-03-08T00:46:30.463ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
13102025-03-08T00:46:30.463ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13112025-03-08T00:46:30.464ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed7c6a9-29b7-4ceb-8c64-8c125477e351", "content-length": "26", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
13122025-03-08T00:46:30.464ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13132025-03-08T00:46:30.465ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
13142025-03-08T00:46:30.465ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13152025-03-08T00:46:30.466ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c071a38-4630-4988-bafa-b00405483a3c", "content-length": "26", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
13162025-03-08T00:46:30.466ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13172025-03-08T00:46:30.466ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13182025-03-08T00:46:30.469ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13192025-03-08T00:46:30.469ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 175.577462ms iteration = 14 status = {"physical_disks_added":0}
13202025-03-08T00:46:30.476ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
13212025-03-08T00:46:30.476ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
13222025-03-08T00:46:30.476ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 183.352774ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
13232025-03-08T00:46:30.494ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
13242025-03-08T00:46:30.494ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 0 version = 2
13252025-03-08T00:46:30.494ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13262025-03-08T00:46:30.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13272025-03-08T00:46:30.497ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13282025-03-08T00:46:30.498ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13292025-03-08T00:46:30.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13302025-03-08T00:46:30.501ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
13312025-03-08T00:46:30.501ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 207.649614ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
13322025-03-08T00:46:30.502ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13332025-03-08T00:46:30.503ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13342025-03-08T00:46:30.503ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13352025-03-08T00:46:30.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13362025-03-08T00:46:30.505ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 1 version = 2
13372025-03-08T00:46:30.505ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
13382025-03-08T00:46:30.505ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-08 00:40:32.180226 UTC dns_group = external generation = 2
13392025-03-08T00:46:30.505ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 212.843072ms iteration = 7 status = {"generation":2}
13402025-03-08T00:46:30.506ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
13412025-03-08T00:46:30.506ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13422025-03-08T00:46:30.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13432025-03-08T00:46:30.507ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 214.117213ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
13442025-03-08T00:46:30.507ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
13452025-03-08T00:46:30.507ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
13462025-03-08T00:46:30.507ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13472025-03-08T00:46:30.508ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13482025-03-08T00:46:30.508ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
13492025-03-08T00:46:30.508ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 214.419081ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
13502025-03-08T00:46:30.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13512025-03-08T00:46:30.508ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 214.949654ms iteration = 13 status = 0
13522025-03-08T00:46:30.509ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13532025-03-08T00:46:30.509ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13542025-03-08T00:46:30.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13552025-03-08T00:46:30.512ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13562025-03-08T00:46:30.512ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13572025-03-08T00:46:30.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13582025-03-08T00:46:30.515ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
13592025-03-08T00:46:30.515ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
13602025-03-08T00:46:30.515ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-08 00:40:30.694594 UTC dns_group = internal generation = 1
13612025-03-08T00:46:30.515ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 222.752086ms iteration = 7 status = {"generation":1}
13622025-03-08T00:46:30.546ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
13632025-03-08T00:46:30.546ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13642025-03-08T00:46:30.546ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "54747797-3a96-44a8-803e-3dc0fd549a94", "content-length": "26", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
13652025-03-08T00:46:30.546ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13662025-03-08T00:46:30.548ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
13672025-03-08T00:46:30.553ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13682025-03-08T00:46:30.555ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13692025-03-08T00:46:30.555ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13702025-03-08T00:46:30.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13712025-03-08T00:46:30.557ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13722025-03-08T00:46:30.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13732025-03-08T00:46:30.606ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-08T00:40:26.976319Z, time_modified: 2025-03-08T00:40:26.976319Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }) }
13742025-03-08T00:46:30.606ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 313.994587ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]}},"default_endpoint":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
13752025-03-08T00:46:30.627ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 334.245568ms iteration = 13 status = {}
13762025-03-08T00:46:30.644ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
13772025-03-08T00:46:30.644ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "6d51c12a-31ff-4c80-ac11-7805535e6776", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
13782025-03-08T00:46:30.644ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
13792025-03-08T00:46:30.646ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "fdf24a53-28a4-464a-b915-84e0860fcd52", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
13802025-03-08T00:46:30.646ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
13812025-03-08T00:46:30.648ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
13822025-03-08T00:46:30.648ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 355.078471ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
13832025-03-08T00:46:30.650ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
13842025-03-08T00:46:30.722ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
13852025-03-08T00:46:30.722ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "165ef39a-90cf-4784-8736-9ad437df3575", "content-length": "1", "date": "Sat, 08 Mar 2025 00:46:29 GMT"} })
13862025-03-08T00:46:30.724ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13872025-03-08T00:46:30.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13882025-03-08T00:46:30.743ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
13892025-03-08T00:46:30.743ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
13902025-03-08T00:46:30.743ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b156ce75-95a5-4fdb-84c4-400347c54b8a", "content-length": "372", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
13912025-03-08T00:46:30.743ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13922025-03-08T00:46:30.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13932025-03-08T00:46:30.745ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13942025-03-08T00:46:30.746ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 453.150991ms iteration = 13 status = null
13952025-03-08T00:46:30.748ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13962025-03-08T00:46:30.748ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13972025-03-08T00:46:30.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13982025-03-08T00:46:30.750ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13992025-03-08T00:46:30.753ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14002025-03-08T00:46:30.753ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14012025-03-08T00:46:30.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14022025-03-08T00:46:30.759ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
14032025-03-08T00:46:30.760ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5d3cd6ad-d94e-4bd5-8548-87c4d31c5656", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
14042025-03-08T00:46:30.760ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
14052025-03-08T00:46:30.761ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "acb4564d-9b72-454f-a9a7-abc15069f445", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
14062025-03-08T00:46:30.761ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
14072025-03-08T00:46:30.763ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "22255380-cf8b-4c02-89b2-0baf48c9184b", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
14082025-03-08T00:46:30.763ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
14092025-03-08T00:46:30.765ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
14102025-03-08T00:46:30.765ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 472.827466ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
14112025-03-08T00:46:30.803ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
14122025-03-08T00:46:30.803ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
14132025-03-08T00:46:30.803ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "686eaccb-1155-4609-8e35-9e56b74f4bb6", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14142025-03-08T00:46:30.803ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14152025-03-08T00:46:30.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14162025-03-08T00:46:30.806ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14172025-03-08T00:46:30.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14182025-03-08T00:46:30.810ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14192025-03-08T00:46:30.810ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14202025-03-08T00:46:30.814ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
14212025-03-08T00:46:30.814ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14222025-03-08T00:46:30.814ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "43e1cd96-a572-4b16-8418-a96de7f3e3b5", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14232025-03-08T00:46:30.816ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 522.953984ms iteration = 13 status = {}
14242025-03-08T00:46:30.817ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
14252025-03-08T00:46:30.817ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0042dbb6-223f-405b-87c5-1960e3168f37", "content-length": "691", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14262025-03-08T00:46:30.817ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
14272025-03-08T00:46:30.817ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
14282025-03-08T00:46:30.817ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
14292025-03-08T00:46:30.819ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 525.325262ms iteration = 13 status = {}
14302025-03-08T00:46:30.841ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14312025-03-08T00:46:30.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14322025-03-08T00:46:30.844ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
14332025-03-08T00:46:30.844ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14342025-03-08T00:46:30.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14352025-03-08T00:46:30.854ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
14362025-03-08T00:46:30.855ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "891c83cb-28c4-4ec4-af80-f66821719c57", "content-length": "134", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14372025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
14382025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
14392025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
14402025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
14412025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14422025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14432025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14442025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14452025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14462025-03-08T00:46:30.855ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14472025-03-08T00:46:30.856ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
14482025-03-08T00:46:30.856ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14492025-03-08T00:46:30.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14502025-03-08T00:46:30.857ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
14512025-03-08T00:46:30.857ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c11a5722-e1bb-4102-adf3-578860135a0d", "content-length": "2", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14522025-03-08T00:46:30.857ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
14532025-03-08T00:46:30.857ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaa7e98e-7eda-4109-a8cd-d4dbb3660abf", "content-length": "52", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14542025-03-08T00:46:30.857ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
14552025-03-08T00:46:30.857ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
14562025-03-08T00:46:30.873ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
14572025-03-08T00:46:30.873ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
14582025-03-08T00:46:30.905ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
14592025-03-08T00:46:30.905ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:38459
14602025-03-08T00:46:30.906ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:38459 req_id = c1706cbe-9d18-4509-8b79-2d8dce2d1149 roles = RoleSet { roles: {} } uri = /metrics/producers
14612025-03-08T00:46:30.906ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:38459 req_id = c1706cbe-9d18-4509-8b79-2d8dce2d1149 resource = Database result = Ok(()) uri = /metrics/producers
14622025-03-08T00:46:30.942ZINFOnexus: assigned collector to new producer collector_id = c5a56912-50aa-493d-9612-f426cf7f1072 file = nexus/src/app/oximeter.rs:76 producer_id = 841ee309-4e1e-443f-ae06-2ad5eb454a8d
14632025-03-08T00:46:30.942ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 36700 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:38459 req_id = c1706cbe-9d18-4509-8b79-2d8dce2d1149 response_code = 201 uri = /metrics/producers
14642025-03-08T00:46:30.942ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "c1706cbe-9d18-4509-8b79-2d8dce2d1149", "content-length": "41", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14652025-03-08T00:46:30.944ZDEBGnexus (producer-registration-task): registered with nexus successfully
14662025-03-08T00:46:30.944ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
14672025-03-08T00:46:30.963ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1ce9aef5-6e19-4b2a-8a37-cb4f6db3b97c", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14682025-03-08T00:46:30.963ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
14692025-03-08T00:46:30.963ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14702025-03-08T00:46:30.967ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "a03d2eff-ff55-4814-ba6b-5f999fe6e441", "content-length": "682", "date": "Sat, 08 Mar 2025 00:46:30 GMT"} })
14712025-03-08T00:46:30.968ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14722025-03-08T00:46:30.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14732025-03-08T00:46:30.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14742025-03-08T00:46:30.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14752025-03-08T00:46:30.976ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14762025-03-08T00:46:30.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14772025-03-08T00:46:30.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14782025-03-08T00:46:30.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14792025-03-08T00:46:30.985ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 691.447232ms iteration = 13 status = {}
14802025-03-08T00:46:32.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 315 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = a973c865-6408-4a20-b653-b708140f7627 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
14812025-03-08T00:46:40.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
14822025-03-08T00:46:40.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14832025-03-08T00:46:40.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14842025-03-08T00:46:40.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
14852025-03-08T00:46:40.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14862025-03-08T00:46:40.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14872025-03-08T00:46:40.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14882025-03-08T00:46:40.295ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14892025-03-08T00:46:40.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14902025-03-08T00:46:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14912025-03-08T00:46:40.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14922025-03-08T00:46:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14932025-03-08T00:46:40.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14942025-03-08T00:46:40.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14952025-03-08T00:46:40.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14962025-03-08T00:46:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
14972025-03-08T00:46:40.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14982025-03-08T00:46:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14992025-03-08T00:46:40.314ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
15002025-03-08T00:46:40.314ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.449659ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
15012025-03-08T00:46:40.325ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.909658ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15022025-03-08T00:46:42.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 370 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = b5b3baff-f5fa-45cd-9b50-1590b24015d4 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
15032025-03-08T00:46:50.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
15042025-03-08T00:46:50.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15052025-03-08T00:46:50.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15062025-03-08T00:46:50.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
15072025-03-08T00:46:50.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15082025-03-08T00:46:50.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15092025-03-08T00:46:50.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15102025-03-08T00:46:50.296ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15112025-03-08T00:46:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15122025-03-08T00:46:50.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15132025-03-08T00:46:50.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15142025-03-08T00:46:50.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15152025-03-08T00:46:50.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15162025-03-08T00:46:50.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15172025-03-08T00:46:50.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15182025-03-08T00:46:50.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
15192025-03-08T00:46:50.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15202025-03-08T00:46:50.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15212025-03-08T00:46:50.317ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
15222025-03-08T00:46:50.317ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.255618ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
15232025-03-08T00:46:50.328ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.886551ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15242025-03-08T00:46:52.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 366 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 7e612c42-ae79-422b-af4c-7c949d587564 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
15252025-03-08T00:46:57.640ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:60683
15262025-03-08T00:46:57.640ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60683 req_id = f20c4305-88df-4739-a3d6-5752d7b923e4 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
15272025-03-08T00:46:57.641ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60683 req_id = f20c4305-88df-4739-a3d6-5752d7b923e4 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
15282025-03-08T00:46:57.643ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2646 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60683 req_id = f20c4305-88df-4739-a3d6-5752d7b923e4 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
15292025-03-08T00:46:57.643ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60683 req_id = 27cfbd65-a40a-4fc1-930e-962b2fc41c03 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
15302025-03-08T00:46:57.644ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60683 req_id = 27cfbd65-a40a-4fc1-930e-962b2fc41c03 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
15312025-03-08T00:46:57.646ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2605 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60683 req_id = 27cfbd65-a40a-4fc1-930e-962b2fc41c03 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
15322025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
15332025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15342025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
15352025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
15362025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
15372025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
15382025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15392025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15402025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15412025-03-08T00:47:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
15422025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
15432025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
15442025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
15452025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15462025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15472025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
15482025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15492025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
15502025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
15512025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
15522025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
15532025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
15542025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
15552025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15562025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
15572025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
15582025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
15592025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15602025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15612025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
15622025-03-08T00:47:00.294ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
15632025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15642025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
15652025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15662025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
15672025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
15682025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
15692025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
15702025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
15712025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15722025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
15732025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15742025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
15752025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15762025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
15772025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15782025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
15792025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15802025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15812025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15822025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
15832025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15842025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
15852025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15862025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15872025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
15882025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15892025-03-08T00:47:00.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15902025-03-08T00:47:00.296ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1303, tv_nsec: 319173382 } })
15912025-03-08T00:47:00.296ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1303, tv_nsec: 319461029 } })
15922025-03-08T00:47:00.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
15932025-03-08T00:47:00.297ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15942025-03-08T00:47:00.297ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.954347ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
15952025-03-08T00:47:00.297ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
15962025-03-08T00:47:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15972025-03-08T00:47:00.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15982025-03-08T00:47:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15992025-03-08T00:47:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
16002025-03-08T00:47:00.300ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16012025-03-08T00:47:00.300ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16022025-03-08T00:47:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16032025-03-08T00:47:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16042025-03-08T00:47:00.301ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16052025-03-08T00:47:00.303ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16062025-03-08T00:47:00.304ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16072025-03-08T00:47:00.304ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16082025-03-08T00:47:00.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16092025-03-08T00:47:00.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
16102025-03-08T00:47:00.305ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16112025-03-08T00:47:00.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16122025-03-08T00:47:00.309ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16132025-03-08T00:47:00.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16142025-03-08T00:47:00.309ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-03-08T00:47:00.310ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16162025-03-08T00:47:00.312ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
16172025-03-08T00:47:00.312ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16182025-03-08T00:47:00.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16192025-03-08T00:47:00.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16202025-03-08T00:47:00.313ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16212025-03-08T00:47:00.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16222025-03-08T00:47:00.315ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16232025-03-08T00:47:00.315ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16242025-03-08T00:47:00.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16252025-03-08T00:47:00.315ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16262025-03-08T00:47:00.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16272025-03-08T00:47:00.317ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16282025-03-08T00:47:00.317ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16292025-03-08T00:47:00.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16302025-03-08T00:47:00.318ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1303, tv_nsec: 340887217 } })
16312025-03-08T00:47:00.318ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
16322025-03-08T00:47:00.323ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16332025-03-08T00:47:00.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16342025-03-08T00:47:00.324ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
16352025-03-08T00:47:00.324ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 30.38843ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
16362025-03-08T00:47:00.324ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16372025-03-08T00:47:00.327ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16382025-03-08T00:47:00.327ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16392025-03-08T00:47:00.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1640 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
16412025-03-08T00:47:00.341ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 47.282027ms iteration = 15 status = {"physical_disks_added":0}
16422025-03-08T00:47:00.348ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16432025-03-08T00:47:00.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16442025-03-08T00:47:00.353ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 58.976296ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
16452025-03-08T00:47:00.359ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.875341ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
16462025-03-08T00:47:00.359ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16472025-03-08T00:47:00.359ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16482025-03-08T00:47:00.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16492025-03-08T00:47:00.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16502025-03-08T00:47:00.364ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
16512025-03-08T00:47:00.364ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16522025-03-08T00:47:00.364ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3971fdce-0752-4dda-86ba-ce2edecd3c92", "content-length": "26", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
16532025-03-08T00:47:00.364ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16542025-03-08T00:47:00.366ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16552025-03-08T00:47:00.366ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
16562025-03-08T00:47:00.366ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16572025-03-08T00:47:00.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16582025-03-08T00:47:00.366ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9b827f1-e6da-4c73-bdef-9b366fa7b019", "content-length": "26", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
16592025-03-08T00:47:00.366ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16602025-03-08T00:47:00.366ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16612025-03-08T00:47:00.366ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
16622025-03-08T00:47:00.366ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16632025-03-08T00:47:00.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16642025-03-08T00:47:00.367ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16652025-03-08T00:47:00.368ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16662025-03-08T00:47:00.370ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 75.75112ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
16672025-03-08T00:47:00.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16682025-03-08T00:47:00.371ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16692025-03-08T00:47:00.371ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16702025-03-08T00:47:00.371ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
16712025-03-08T00:47:00.371ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16722025-03-08T00:47:00.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16732025-03-08T00:47:00.374ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16742025-03-08T00:47:00.374ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16752025-03-08T00:47:00.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16762025-03-08T00:47:00.375ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16772025-03-08T00:47:00.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16782025-03-08T00:47:00.375ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
16792025-03-08T00:47:00.375ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16802025-03-08T00:47:00.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16812025-03-08T00:47:00.378ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 84.644029ms iteration = 14 status = 0
16822025-03-08T00:47:00.381ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 86.831502ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16832025-03-08T00:47:00.381ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 87.676543ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16842025-03-08T00:47:00.382ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16852025-03-08T00:47:00.386ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
16862025-03-08T00:47:00.386ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16872025-03-08T00:47:00.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16882025-03-08T00:47:00.398ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16892025-03-08T00:47:00.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16902025-03-08T00:47:00.401ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
16912025-03-08T00:47:00.401ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 106.919808ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
16922025-03-08T00:47:00.401ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 108.000425ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
16932025-03-08T00:47:00.408ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 114.103445ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16942025-03-08T00:47:00.412ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16952025-03-08T00:47:00.415ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16962025-03-08T00:47:00.415ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16972025-03-08T00:47:00.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16982025-03-08T00:47:00.418ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16992025-03-08T00:47:00.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17002025-03-08T00:47:00.428ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17012025-03-08T00:47:00.428ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
17022025-03-08T00:47:00.428ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 133.918835ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
17032025-03-08T00:47:00.428ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17042025-03-08T00:47:00.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17052025-03-08T00:47:00.430ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17062025-03-08T00:47:00.431ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
17072025-03-08T00:47:00.431ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
17082025-03-08T00:47:00.431ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8bdf181-c8b6-415c-b51a-afeeeed29388", "content-length": "26", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
17092025-03-08T00:47:00.431ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
17102025-03-08T00:47:00.432ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17112025-03-08T00:47:00.432ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17122025-03-08T00:47:00.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17132025-03-08T00:47:00.433ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
17142025-03-08T00:47:00.434ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17152025-03-08T00:47:00.437ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17162025-03-08T00:47:00.437ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17172025-03-08T00:47:00.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17182025-03-08T00:47:00.450ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
17192025-03-08T00:47:00.450ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 156.183393ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
17202025-03-08T00:47:00.463ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
17212025-03-08T00:47:00.463ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "56c2883b-0661-43b9-84ae-9d8589e87eb8", "content-length": "1", "date": "Sat, 08 Mar 2025 00:46:59 GMT"} })
17222025-03-08T00:47:00.465ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17232025-03-08T00:47:00.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17242025-03-08T00:47:00.467ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 174.833794ms iteration = 14 status = null
17252025-03-08T00:47:00.471ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 178.255129ms iteration = 14 status = {}
17262025-03-08T00:47:00.527ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
17272025-03-08T00:47:00.527ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1e9b90a-489f-4a30-977b-25e032aca4e1", "content-length": "691", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
17282025-03-08T00:47:00.527ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
17292025-03-08T00:47:00.527ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
17302025-03-08T00:47:00.527ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
17312025-03-08T00:47:00.528ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 234.436285ms iteration = 14 status = {}
17322025-03-08T00:47:00.540ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
17332025-03-08T00:47:00.540ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17342025-03-08T00:47:00.540ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4103dc59-eb03-4511-bfa7-e238e9d19acd", "content-length": "372", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
17352025-03-08T00:47:00.540ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17362025-03-08T00:47:00.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17372025-03-08T00:47:00.555ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
17382025-03-08T00:47:00.555ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
17392025-03-08T00:47:00.555ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ca5fd605-5ca3-4310-ba92-97209dfad0b5", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
17402025-03-08T00:47:00.555ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17412025-03-08T00:47:00.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17422025-03-08T00:47:00.558ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17432025-03-08T00:47:00.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17442025-03-08T00:47:00.561ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17452025-03-08T00:47:00.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17462025-03-08T00:47:00.564ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
17472025-03-08T00:47:00.564ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17482025-03-08T00:47:00.564ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0e5709fa-b57d-47a9-8310-d43a154d1832", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
17492025-03-08T00:47:00.566ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 271.846517ms iteration = 14 status = {}
17502025-03-08T00:47:00.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17512025-03-08T00:47:00.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17522025-03-08T00:47:00.595ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
17532025-03-08T00:47:00.595ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17542025-03-08T00:47:00.596ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17552025-03-08T00:47:00.605ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
17562025-03-08T00:47:00.605ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "278eb78a-1132-45f2-99e3-7b0c75734503", "content-length": "134", "date": "Sat, 08 Mar 2025 00:46:59 GMT"} })
17572025-03-08T00:47:00.605ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
17582025-03-08T00:47:00.605ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
17592025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
17602025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
17612025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17622025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17632025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17642025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17652025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17662025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17672025-03-08T00:47:00.606ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
17682025-03-08T00:47:00.606ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17692025-03-08T00:47:00.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17702025-03-08T00:47:00.608ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
17712025-03-08T00:47:00.609ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca9f49b1-720c-4b63-a627-2d7f4267bb6b", "content-length": "2", "date": "Sat, 08 Mar 2025 00:46:59 GMT"} })
17722025-03-08T00:47:00.609ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
17732025-03-08T00:47:00.609ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "a97c7306-7fd6-42f5-9bee-6e26e7d7fbd8", "content-length": "52", "date": "Sat, 08 Mar 2025 00:46:59 GMT"} })
17742025-03-08T00:47:00.609ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
17752025-03-08T00:47:00.609ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
17762025-03-08T00:47:00.735ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "910563dc-ce14-4ba5-a438-4fe597323471", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
17772025-03-08T00:47:00.735ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
17782025-03-08T00:47:00.735ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
17792025-03-08T00:47:00.736ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8985b743-8e58-4bce-bbb7-c51fccb5b3ee", "content-length": "682", "date": "Sat, 08 Mar 2025 00:47:00 GMT"} })
17802025-03-08T00:47:00.736ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17812025-03-08T00:47:00.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17822025-03-08T00:47:00.741ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17832025-03-08T00:47:00.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17842025-03-08T00:47:00.744ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17852025-03-08T00:47:00.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17862025-03-08T00:47:00.747ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17872025-03-08T00:47:00.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17882025-03-08T00:47:00.755ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 460.899192ms iteration = 14 status = {}
17892025-03-08T00:47:02.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 366 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 3520bfb5-749d-47a6-b8c3-adeb39255272 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
17902025-03-08T00:47:10.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
17912025-03-08T00:47:10.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17922025-03-08T00:47:10.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17932025-03-08T00:47:10.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
17942025-03-08T00:47:10.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17952025-03-08T00:47:10.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17962025-03-08T00:47:10.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17972025-03-08T00:47:10.295ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17982025-03-08T00:47:10.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17992025-03-08T00:47:10.297ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.943557ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18002025-03-08T00:47:10.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18012025-03-08T00:47:10.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18022025-03-08T00:47:10.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18032025-03-08T00:47:10.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18042025-03-08T00:47:10.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18052025-03-08T00:47:10.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18062025-03-08T00:47:10.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
18072025-03-08T00:47:10.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18082025-03-08T00:47:10.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18092025-03-08T00:47:10.317ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
18102025-03-08T00:47:10.317ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.336491ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
18112025-03-08T00:47:12.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 336 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = e0402bf1-a958-4847-bfbd-8973adfe8e2e response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
18122025-03-08T00:47:20.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
18132025-03-08T00:47:20.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18142025-03-08T00:47:20.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18152025-03-08T00:47:20.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
18162025-03-08T00:47:20.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18172025-03-08T00:47:20.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18182025-03-08T00:47:20.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18192025-03-08T00:47:20.295ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18202025-03-08T00:47:20.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18212025-03-08T00:47:20.297ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.903977ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18222025-03-08T00:47:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18232025-03-08T00:47:20.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18242025-03-08T00:47:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18252025-03-08T00:47:20.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18262025-03-08T00:47:20.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18272025-03-08T00:47:20.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18282025-03-08T00:47:20.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
18292025-03-08T00:47:20.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18302025-03-08T00:47:20.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18312025-03-08T00:47:20.314ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
18322025-03-08T00:47:20.314ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.570722ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
18332025-03-08T00:47:22.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 329 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 5a7c1b9a-74c8-4322-9c85-9af2ca868452 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
18342025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
18352025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
18362025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
18372025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
18382025-03-08T00:47:30.292ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-08 00:37:30.292883138 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
18392025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
18402025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
18412025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
18422025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18432025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
18442025-03-08T00:47:30.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18452025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
18462025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
18472025-03-08T00:47:30.293ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = a346f089-8033-4b92-b854-361d1da5cab2
18482025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 15.871\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"a346f089-8033-4b92-b854-361d1da5cab2"}
18492025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
18502025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
18512025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
18522025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18532025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18542025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
18552025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18562025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18572025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18582025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
18592025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
18602025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
18612025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
18622025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
18632025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
18642025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
18652025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
18662025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18672025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18682025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18692025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
18702025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18712025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
18722025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18732025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
18742025-03-08T00:47:30.293ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
18752025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
18762025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18772025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18782025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
18792025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
18802025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
18812025-03-08T00:47:30.293ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
18822025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
18832025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
18842025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
18852025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
18862025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18872025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
18882025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18892025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
18902025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18912025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
18922025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
18932025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
18942025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
18952025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
18962025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
18972025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
18982025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18992025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
19002025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
19012025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19022025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19032025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19042025-03-08T00:47:30.293ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("59287b7e-0d61-4586-b7c5-e79476927737.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1333, tv_nsec: 316406604 } })
19052025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19062025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19072025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19082025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19092025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19102025-03-08T00:47:30.293ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1ac98a83-1243-46f5-99db-fd29222026d8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("508de01a-93ec-4309-a59e-0cfd22b5027a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1333, tv_nsec: 316575408 } })
19112025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19122025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
19132025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19142025-03-08T00:47:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19152025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19162025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19172025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
19182025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
19192025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19202025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
19212025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19222025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
19232025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19242025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
19252025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19262025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
19272025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
19282025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.187119ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
19292025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19302025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
19312025-03-08T00:47:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.206969ms iteration = 8 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
19322025-03-08T00:47:30.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19332025-03-08T00:47:30.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19342025-03-08T00:47:30.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1333, tv_nsec: 318630428 } })
19352025-03-08T00:47:30.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
19362025-03-08T00:47:30.296ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-08 00:37:30.292883138 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
19372025-03-08T00:47:30.296ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.63952ms iteration = 8 status = {"expiration":"2025-03-08T00:37:30.292883138Z","pruned":[]}
19382025-03-08T00:47:30.296ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.766843ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
19392025-03-08T00:47:30.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19402025-03-08T00:47:30.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19412025-03-08T00:47:30.297ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19422025-03-08T00:47:30.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19432025-03-08T00:47:30.298ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1333, tv_nsec: 321387047 } })
19442025-03-08T00:47:30.298ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
19452025-03-08T00:47:30.300ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19462025-03-08T00:47:30.300ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19472025-03-08T00:47:30.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19482025-03-08T00:47:30.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19492025-03-08T00:47:30.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19502025-03-08T00:47:30.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19512025-03-08T00:47:30.303ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19522025-03-08T00:47:30.306ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
19532025-03-08T00:47:30.306ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19542025-03-08T00:47:30.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19552025-03-08T00:47:30.307ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 13.919314ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
19562025-03-08T00:47:30.307ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
19572025-03-08T00:47:30.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19582025-03-08T00:47:30.308ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19592025-03-08T00:47:30.308ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19602025-03-08T00:47:30.310ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
19612025-03-08T00:47:30.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
19622025-03-08T00:47:30.311ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
19632025-03-08T00:47:30.311ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19642025-03-08T00:47:30.311ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19652025-03-08T00:47:30.311ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19662025-03-08T00:47:30.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19672025-03-08T00:47:30.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19682025-03-08T00:47:30.313ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19692025-03-08T00:47:30.313ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.408244ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
19702025-03-08T00:47:30.313ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19712025-03-08T00:47:30.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19722025-03-08T00:47:30.314ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19732025-03-08T00:47:30.316ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
19742025-03-08T00:47:30.316ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
19752025-03-08T00:47:30.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
19762025-03-08T00:47:30.316ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19772025-03-08T00:47:30.316ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19782025-03-08T00:47:30.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19792025-03-08T00:47:30.319ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19802025-03-08T00:47:30.319ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19812025-03-08T00:47:30.319ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19822025-03-08T00:47:30.320ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19832025-03-08T00:47:30.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19842025-03-08T00:47:30.320ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19852025-03-08T00:47:30.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19862025-03-08T00:47:30.322ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
19872025-03-08T00:47:30.322ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19882025-03-08T00:47:30.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19892025-03-08T00:47:30.324ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19902025-03-08T00:47:30.324ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19912025-03-08T00:47:30.325ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 32.028ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19922025-03-08T00:47:30.325ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19932025-03-08T00:47:30.325ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19942025-03-08T00:47:30.325ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19952025-03-08T00:47:30.325ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19962025-03-08T00:47:30.325ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19972025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19982025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19992025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
20002025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20012025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20022025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20032025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20042025-03-08T00:47:30.326ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20052025-03-08T00:47:30.328ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1333, tv_nsec: 351196852 } })
20062025-03-08T00:47:30.328ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20072025-03-08T00:47:30.328ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20082025-03-08T00:47:30.328ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
20092025-03-08T00:47:30.328ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
20102025-03-08T00:47:30.329ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 35.523236ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
20112025-03-08T00:47:30.329ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
20122025-03-08T00:47:30.329ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20132025-03-08T00:47:30.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20142025-03-08T00:47:30.330ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20152025-03-08T00:47:30.330ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
20162025-03-08T00:47:30.330ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20172025-03-08T00:47:30.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20182025-03-08T00:47:30.333ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
20192025-03-08T00:47:30.333ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20202025-03-08T00:47:30.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20212025-03-08T00:47:30.334ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
20222025-03-08T00:47:30.334ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20232025-03-08T00:47:30.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20242025-03-08T00:47:30.334ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
20252025-03-08T00:47:30.334ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 41.434932ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
20262025-03-08T00:47:30.354ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20272025-03-08T00:47:30.354ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20282025-03-08T00:47:30.357ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20292025-03-08T00:47:30.357ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
20302025-03-08T00:47:30.357ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20312025-03-08T00:47:30.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20322025-03-08T00:47:30.359ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20332025-03-08T00:47:30.359ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20342025-03-08T00:47:30.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20352025-03-08T00:47:30.359ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20362025-03-08T00:47:30.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20372025-03-08T00:47:30.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20382025-03-08T00:47:30.365ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2039 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
20402025-03-08T00:47:30.368ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20412025-03-08T00:47:30.369ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20422025-03-08T00:47:30.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20432025-03-08T00:47:30.372ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20442025-03-08T00:47:30.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20452025-03-08T00:47:30.374ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20462025-03-08T00:47:30.375ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
20472025-03-08T00:47:30.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
20482025-03-08T00:47:30.376ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
20492025-03-08T00:47:30.376ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 83.803558ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
20502025-03-08T00:47:30.377ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 84.585857ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
20512025-03-08T00:47:30.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20522025-03-08T00:47:30.378ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20532025-03-08T00:47:30.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20542025-03-08T00:47:30.379ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 86.077004ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
20552025-03-08T00:47:30.382ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
20562025-03-08T00:47:30.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
20572025-03-08T00:47:30.383ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
20582025-03-08T00:47:30.383ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
20592025-03-08T00:47:30.383ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 90.000861ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
20602025-03-08T00:47:30.384ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20612025-03-08T00:47:30.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20622025-03-08T00:47:30.386ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 92.595344ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20632025-03-08T00:47:30.386ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 92.668687ms iteration = 16 status = {"physical_disks_added":0}
20642025-03-08T00:47:30.390ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20652025-03-08T00:47:30.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20662025-03-08T00:47:30.393ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20672025-03-08T00:47:30.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20682025-03-08T00:47:30.394ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20692025-03-08T00:47:30.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20702025-03-08T00:47:30.395ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20712025-03-08T00:47:30.395ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20722025-03-08T00:47:30.399ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
20732025-03-08T00:47:30.399ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20742025-03-08T00:47:30.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20752025-03-08T00:47:30.399ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
20762025-03-08T00:47:30.399ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
20772025-03-08T00:47:30.399ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 106.170699ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
20782025-03-08T00:47:30.400ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
20792025-03-08T00:47:30.400ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20802025-03-08T00:47:30.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20812025-03-08T00:47:30.405ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 111.884631ms iteration = 15 status = 0
20822025-03-08T00:47:30.408ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20832025-03-08T00:47:30.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20842025-03-08T00:47:30.416ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20852025-03-08T00:47:30.419ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
20862025-03-08T00:47:30.419ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20872025-03-08T00:47:30.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20882025-03-08T00:47:30.420ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20892025-03-08T00:47:30.420ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 0 version = 2
20902025-03-08T00:47:30.420ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20912025-03-08T00:47:30.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20922025-03-08T00:47:30.421ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20932025-03-08T00:47:30.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20942025-03-08T00:47:30.422ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20952025-03-08T00:47:30.423ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20962025-03-08T00:47:30.423ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20972025-03-08T00:47:30.425ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20982025-03-08T00:47:30.425ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
20992025-03-08T00:47:30.425ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 132.635623ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
21002025-03-08T00:47:30.426ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
21012025-03-08T00:47:30.426ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21022025-03-08T00:47:30.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21032025-03-08T00:47:30.427ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
21042025-03-08T00:47:30.427ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21052025-03-08T00:47:30.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21062025-03-08T00:47:30.429ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21072025-03-08T00:47:30.429ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21082025-03-08T00:47:30.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21092025-03-08T00:47:30.431ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
21102025-03-08T00:47:30.431ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
21112025-03-08T00:47:30.431ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21122025-03-08T00:47:30.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21132025-03-08T00:47:30.433ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21142025-03-08T00:47:30.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21152025-03-08T00:47:30.434ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 1 version = 2
21162025-03-08T00:47:30.435ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
21172025-03-08T00:47:30.435ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-08 00:40:32.180226 UTC dns_group = external generation = 2
21182025-03-08T00:47:30.435ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 141.982704ms iteration = 8 status = {"generation":2}
21192025-03-08T00:47:30.435ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 141.910162ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
21202025-03-08T00:47:30.435ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21212025-03-08T00:47:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21222025-03-08T00:47:30.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21232025-03-08T00:47:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21242025-03-08T00:47:30.448ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
21252025-03-08T00:47:30.448ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
21262025-03-08T00:47:30.448ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-08 00:40:30.694594 UTC dns_group = internal generation = 1
21272025-03-08T00:47:30.448ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 155.966338ms iteration = 8 status = {"generation":1}
21282025-03-08T00:47:30.457ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
21292025-03-08T00:47:30.457ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 164.417767ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
21302025-03-08T00:47:30.479ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
21312025-03-08T00:47:30.479ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21322025-03-08T00:47:30.480ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a7bc198-a6e5-4d41-ba11-b1980258b8b7", "content-length": "26", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
21332025-03-08T00:47:30.480ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21342025-03-08T00:47:30.481ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
21352025-03-08T00:47:30.481ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 188.47875ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
21362025-03-08T00:47:30.483ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
21372025-03-08T00:47:30.483ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
21382025-03-08T00:47:30.486ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-08T00:40:26.976319Z, time_modified: 2025-03-08T00:40:26.976319Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }) }
21392025-03-08T00:47:30.486ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 193.657819ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]}},"default_endpoint":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
21402025-03-08T00:47:30.563ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
21412025-03-08T00:47:30.563ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
21422025-03-08T00:47:30.563ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d00f689-8204-4d62-927c-ff2a65d67438", "content-length": "372", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
21432025-03-08T00:47:30.563ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21442025-03-08T00:47:30.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21452025-03-08T00:47:30.606ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
21462025-03-08T00:47:30.606ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21472025-03-08T00:47:30.606ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99ef502-ce3e-4acf-a362-5ce90ed34a7f", "content-length": "26", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
21482025-03-08T00:47:30.606ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21492025-03-08T00:47:30.608ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21502025-03-08T00:47:30.620ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
21512025-03-08T00:47:30.620ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21522025-03-08T00:47:30.621ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0543a817-b6c1-4740-b5dd-a8ce46c090f3", "content-length": "26", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
21532025-03-08T00:47:30.621ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21542025-03-08T00:47:30.622ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21552025-03-08T00:47:30.625ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
21562025-03-08T00:47:30.625ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
21572025-03-08T00:47:30.625ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "0052963e-cb05-4232-96d5-387c52ec5337", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
21582025-03-08T00:47:30.625ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21592025-03-08T00:47:30.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21602025-03-08T00:47:30.629ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21612025-03-08T00:47:30.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21622025-03-08T00:47:30.676ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
21632025-03-08T00:47:30.677ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "5a904d36-21f4-46e6-8606-d8fb68b26635", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21642025-03-08T00:47:30.677ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21652025-03-08T00:47:30.678ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d4f9098a-e29f-4077-b523-61dda84801bc", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21662025-03-08T00:47:30.678ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21672025-03-08T00:47:30.680ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
21682025-03-08T00:47:30.681ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 388.138248ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
21692025-03-08T00:47:30.793ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
21702025-03-08T00:47:30.799ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
21712025-03-08T00:47:30.800ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fb9ec91-a084-424f-997f-a178e4c5b71b", "content-length": "1", "date": "Sat, 08 Mar 2025 00:47:29 GMT"} })
21722025-03-08T00:47:30.803ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21732025-03-08T00:47:30.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21742025-03-08T00:47:30.810ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
21752025-03-08T00:47:30.810ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ecdf06-36bd-41b1-849b-18ab41a2ad53", "content-length": "691", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
21762025-03-08T00:47:30.811ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
21772025-03-08T00:47:30.811ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
21782025-03-08T00:47:30.811ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
21792025-03-08T00:47:30.812ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 519.482508ms iteration = 15 status = {}
21802025-03-08T00:47:30.816ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 523.395824ms iteration = 15 status = null
21812025-03-08T00:47:30.818ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21822025-03-08T00:47:30.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21832025-03-08T00:47:30.821ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
21842025-03-08T00:47:30.821ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
21852025-03-08T00:47:30.822ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "db2c9cab-6b0f-4b92-a075-9e891a494d80", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
21862025-03-08T00:47:30.824ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 530.888989ms iteration = 15 status = {}
21872025-03-08T00:47:30.872ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 580.012902ms iteration = 15 status = {}
21882025-03-08T00:47:30.898ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
21892025-03-08T00:47:30.898ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5242a3c0-2de8-48c2-8b6b-ebe3aea1bcf2", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
21902025-03-08T00:47:30.898ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
21912025-03-08T00:47:30.900ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "b1557c63-49ba-4c3a-895e-6231b8359cb7", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
21922025-03-08T00:47:30.900ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
21932025-03-08T00:47:30.902ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "4536fcf3-9b53-4952-8c85-fef19a0e022f", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
21942025-03-08T00:47:30.902ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
21952025-03-08T00:47:30.904ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
21962025-03-08T00:47:30.904ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 611.959049ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
21972025-03-08T00:47:30.929ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21982025-03-08T00:47:30.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21992025-03-08T00:47:30.933ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
22002025-03-08T00:47:30.933ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22012025-03-08T00:47:30.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22022025-03-08T00:47:30.943ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
22032025-03-08T00:47:30.943ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaf7bdef-778c-458e-8e53-37eabe2cc174", "content-length": "134", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
22042025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
22052025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
22062025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
22072025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
22082025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22092025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22102025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22112025-03-08T00:47:30.943ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22122025-03-08T00:47:30.944ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22132025-03-08T00:47:30.944ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22142025-03-08T00:47:30.944ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
22152025-03-08T00:47:30.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22162025-03-08T00:47:30.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22172025-03-08T00:47:30.947ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
22182025-03-08T00:47:30.947ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "806553e8-09f2-439c-8b7c-325997a8bef2", "content-length": "2", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
22192025-03-08T00:47:30.947ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
22202025-03-08T00:47:30.947ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "24bcab90-3da2-4f3f-a7aa-41d467943cf0", "content-length": "52", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
22212025-03-08T00:47:30.947ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
22222025-03-08T00:47:30.947ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
22232025-03-08T00:47:31.134ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "cc0fe609-0b71-44fc-8d5b-518a5c544231", "date": "Sat, 08 Mar 2025 00:47:30 GMT"} })
22242025-03-08T00:47:31.134ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
22252025-03-08T00:47:31.134ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
22262025-03-08T00:47:31.135ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "4899cf0a-c218-4fc5-b16d-c481916879ea", "content-length": "682", "date": "Sat, 08 Mar 2025 00:47:31 GMT"} })
22272025-03-08T00:47:31.135ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22282025-03-08T00:47:31.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22292025-03-08T00:47:31.139ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22302025-03-08T00:47:31.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22312025-03-08T00:47:31.142ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22322025-03-08T00:47:31.142ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22332025-03-08T00:47:31.144ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22342025-03-08T00:47:31.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22352025-03-08T00:47:31.151ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 858.242407ms iteration = 15 status = {}
22362025-03-08T00:47:32.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 16e25162-fab6-4019-93f4-5df74a31d305 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
22372025-03-08T00:47:40.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
22382025-03-08T00:47:40.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22392025-03-08T00:47:40.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22402025-03-08T00:47:40.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
22412025-03-08T00:47:40.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22422025-03-08T00:47:40.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22432025-03-08T00:47:40.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22442025-03-08T00:47:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22452025-03-08T00:47:40.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22462025-03-08T00:47:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22472025-03-08T00:47:40.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22482025-03-08T00:47:40.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22492025-03-08T00:47:40.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22502025-03-08T00:47:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
22512025-03-08T00:47:40.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22522025-03-08T00:47:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22532025-03-08T00:47:40.316ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
22542025-03-08T00:47:40.316ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.758126ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
22552025-03-08T00:47:40.328ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22562025-03-08T00:47:40.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22572025-03-08T00:47:40.330ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.829058ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22582025-03-08T00:47:42.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 374 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 5b8079f0-b633-4db0-8eec-2df8292ba30c response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
22592025-03-08T00:47:50.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
22602025-03-08T00:47:50.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22612025-03-08T00:47:50.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22622025-03-08T00:47:50.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
22632025-03-08T00:47:50.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22642025-03-08T00:47:50.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22652025-03-08T00:47:50.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22662025-03-08T00:47:50.296ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22672025-03-08T00:47:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22682025-03-08T00:47:50.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22692025-03-08T00:47:50.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22702025-03-08T00:47:50.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22712025-03-08T00:47:50.298ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.389769ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22722025-03-08T00:47:50.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22732025-03-08T00:47:50.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22742025-03-08T00:47:50.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22752025-03-08T00:47:50.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
22762025-03-08T00:47:50.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22772025-03-08T00:47:50.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22782025-03-08T00:47:50.321ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
22792025-03-08T00:47:50.321ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.528094ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
22802025-03-08T00:47:52.651ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = f2dbad39-a9dc-4dcf-811d-ac057bf52d6c response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
22812025-03-08T00:47:57.640ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:45728
22822025-03-08T00:47:57.640ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45728 req_id = f3eb6561-f0b8-4f5a-b11f-79d80a56a87a roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
22832025-03-08T00:47:57.640ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45728 req_id = f3eb6561-f0b8-4f5a-b11f-79d80a56a87a resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
22842025-03-08T00:47:57.642ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 1969 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45728 req_id = f3eb6561-f0b8-4f5a-b11f-79d80a56a87a response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
22852025-03-08T00:47:57.642ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45728 req_id = c48a7607-7a85-4fa5-9b04-650cf2a916dd roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
22862025-03-08T00:47:57.642ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45728 req_id = c48a7607-7a85-4fa5-9b04-650cf2a916dd resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
22872025-03-08T00:47:57.644ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2277 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45728 req_id = c48a7607-7a85-4fa5-9b04-650cf2a916dd response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
22882025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
22892025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
22902025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
22912025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
22922025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22932025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22942025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
22952025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
22962025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22972025-03-08T00:48:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22982025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
22992025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
23002025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
23012025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
23022025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23032025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23042025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
23052025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23062025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
23072025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23082025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23092025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
23102025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23112025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
23122025-03-08T00:48:00.293ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
23132025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23142025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
23152025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23162025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
23172025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
23182025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
23192025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
23202025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
23212025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23222025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
23232025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
23242025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
23252025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23262025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
23272025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
23282025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
23292025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23302025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23312025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23322025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23332025-03-08T00:48:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23342025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23352025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23362025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23372025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
23382025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23392025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23402025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23412025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
23422025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
23432025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
23442025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23452025-03-08T00:48:00.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23462025-03-08T00:48:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1363, tv_nsec: 318337651 } })
23472025-03-08T00:48:00.295ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.106276ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
23482025-03-08T00:48:00.295ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
23492025-03-08T00:48:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1363, tv_nsec: 318804343 } })
23502025-03-08T00:48:00.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
23512025-03-08T00:48:00.296ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23522025-03-08T00:48:00.296ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23532025-03-08T00:48:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23542025-03-08T00:48:00.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23552025-03-08T00:48:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23562025-03-08T00:48:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
23572025-03-08T00:48:00.299ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23582025-03-08T00:48:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
23592025-03-08T00:48:00.299ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23602025-03-08T00:48:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23612025-03-08T00:48:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23622025-03-08T00:48:00.300ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23632025-03-08T00:48:00.301ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23642025-03-08T00:48:00.301ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23652025-03-08T00:48:00.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23662025-03-08T00:48:00.303ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23672025-03-08T00:48:00.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
23682025-03-08T00:48:00.303ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23692025-03-08T00:48:00.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23702025-03-08T00:48:00.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23712025-03-08T00:48:00.304ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23722025-03-08T00:48:00.304ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23732025-03-08T00:48:00.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23742025-03-08T00:48:00.305ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23752025-03-08T00:48:00.306ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23762025-03-08T00:48:00.306ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 13.120904ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
23772025-03-08T00:48:00.307ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 13.647197ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
23782025-03-08T00:48:00.308ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23792025-03-08T00:48:00.308ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23802025-03-08T00:48:00.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23812025-03-08T00:48:00.310ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23822025-03-08T00:48:00.310ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23832025-03-08T00:48:00.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23842025-03-08T00:48:00.313ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1363, tv_nsec: 336609062 } })
23852025-03-08T00:48:00.313ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
23862025-03-08T00:48:00.316ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23872025-03-08T00:48:00.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23882025-03-08T00:48:00.317ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
23892025-03-08T00:48:00.317ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 23.596583ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
23902025-03-08T00:48:00.317ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23912025-03-08T00:48:00.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23922025-03-08T00:48:00.319ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23932025-03-08T00:48:00.320ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23942025-03-08T00:48:00.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23952025-03-08T00:48:00.321ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23962025-03-08T00:48:00.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23972025-03-08T00:48:00.322ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23982025-03-08T00:48:00.322ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23992025-03-08T00:48:00.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2400 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
24012025-03-08T00:48:00.344ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24022025-03-08T00:48:00.344ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
24032025-03-08T00:48:00.344ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24042025-03-08T00:48:00.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24052025-03-08T00:48:00.356ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24062025-03-08T00:48:00.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24072025-03-08T00:48:00.357ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 64.317207ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
24082025-03-08T00:48:00.360ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24092025-03-08T00:48:00.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24102025-03-08T00:48:00.365ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24112025-03-08T00:48:00.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
24122025-03-08T00:48:00.370ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24132025-03-08T00:48:00.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24142025-03-08T00:48:00.373ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 80.136817ms iteration = 16 status = 0
24152025-03-08T00:48:00.373ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24162025-03-08T00:48:00.376ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24172025-03-08T00:48:00.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24182025-03-08T00:48:00.377ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 84.404062ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24192025-03-08T00:48:00.378ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 84.673979ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24202025-03-08T00:48:00.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
24212025-03-08T00:48:00.378ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24222025-03-08T00:48:00.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24232025-03-08T00:48:00.379ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24242025-03-08T00:48:00.380ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24252025-03-08T00:48:00.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24262025-03-08T00:48:00.381ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24272025-03-08T00:48:00.382ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
24282025-03-08T00:48:00.382ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24292025-03-08T00:48:00.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
24302025-03-08T00:48:00.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24312025-03-08T00:48:00.387ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24322025-03-08T00:48:00.390ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
24332025-03-08T00:48:00.390ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24342025-03-08T00:48:00.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24352025-03-08T00:48:00.394ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
24362025-03-08T00:48:00.394ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 102.195792ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
24372025-03-08T00:48:00.398ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24382025-03-08T00:48:00.398ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
24392025-03-08T00:48:00.398ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24402025-03-08T00:48:00.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24412025-03-08T00:48:00.400ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
24422025-03-08T00:48:00.400ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24432025-03-08T00:48:00.400ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9812b19a-216a-40cc-9104-f981137b295f", "content-length": "26", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
24442025-03-08T00:48:00.400ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24452025-03-08T00:48:00.401ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24462025-03-08T00:48:00.412ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
24472025-03-08T00:48:00.412ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24482025-03-08T00:48:00.413ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "05ee69f6-bc5a-4a43-bf01-b3b42559b247", "content-length": "26", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
24492025-03-08T00:48:00.413ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24502025-03-08T00:48:00.415ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24512025-03-08T00:48:00.417ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 123.901587ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
24522025-03-08T00:48:00.421ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
24532025-03-08T00:48:00.421ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24542025-03-08T00:48:00.421ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "451610a9-8044-49c2-8376-4ace0d3acd33", "content-length": "26", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
24552025-03-08T00:48:00.421ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24562025-03-08T00:48:00.422ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
24572025-03-08T00:48:00.422ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 129.026343ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
24582025-03-08T00:48:00.423ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
24592025-03-08T00:48:00.424ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 131.351841ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24602025-03-08T00:48:00.436ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24612025-03-08T00:48:00.436ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
24622025-03-08T00:48:00.436ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 142.727612ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
24632025-03-08T00:48:00.450ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24642025-03-08T00:48:00.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24652025-03-08T00:48:00.459ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24662025-03-08T00:48:00.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24672025-03-08T00:48:00.461ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24682025-03-08T00:48:00.463ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24692025-03-08T00:48:00.463ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24702025-03-08T00:48:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24712025-03-08T00:48:00.464ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24722025-03-08T00:48:00.466ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
24732025-03-08T00:48:00.466ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24742025-03-08T00:48:00.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24752025-03-08T00:48:00.499ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 205.590703ms iteration = 17 status = {"physical_disks_added":0}
24762025-03-08T00:48:00.505ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
24772025-03-08T00:48:00.506ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4893bde4-ecf9-462d-8ad2-1cedc6d7044a", "content-length": "1", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
24782025-03-08T00:48:00.507ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24792025-03-08T00:48:00.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24802025-03-08T00:48:00.522ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
24812025-03-08T00:48:00.522ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24822025-03-08T00:48:00.523ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ae7ab35-6089-40f8-a3c3-5bceec90b12d", "content-length": "372", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
24832025-03-08T00:48:00.523ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24842025-03-08T00:48:00.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24852025-03-08T00:48:00.525ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 233.531072ms iteration = 16 status = null
24862025-03-08T00:48:00.527ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 235.017009ms iteration = 16 status = {}
24872025-03-08T00:48:00.543ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
24882025-03-08T00:48:00.543ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
24892025-03-08T00:48:00.543ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "959d8bf1-0472-4bb6-9b88-6e8ff40c713f", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
24902025-03-08T00:48:00.543ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24912025-03-08T00:48:00.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24922025-03-08T00:48:00.545ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24932025-03-08T00:48:00.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24942025-03-08T00:48:00.548ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24952025-03-08T00:48:00.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24962025-03-08T00:48:00.552ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
24972025-03-08T00:48:00.552ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24982025-03-08T00:48:00.553ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "dace42da-09ff-4419-8d23-4de5f56771a2", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
24992025-03-08T00:48:00.554ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 260.920638ms iteration = 16 status = {}
25002025-03-08T00:48:00.594ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25012025-03-08T00:48:00.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25022025-03-08T00:48:00.597ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
25032025-03-08T00:48:00.597ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25042025-03-08T00:48:00.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25052025-03-08T00:48:00.607ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
25062025-03-08T00:48:00.607ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bff0131-3046-4973-b2a8-972ca028e5fb", "content-length": "134", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
25072025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
25082025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
25092025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
25102025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
25112025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25122025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25132025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25142025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25152025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25162025-03-08T00:48:00.607ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
25172025-03-08T00:48:00.608ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
25182025-03-08T00:48:00.608ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25192025-03-08T00:48:00.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25202025-03-08T00:48:00.610ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
25212025-03-08T00:48:00.610ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "551d9771-2451-43ee-95d9-d7a873822d03", "content-length": "2", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
25222025-03-08T00:48:00.610ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
25232025-03-08T00:48:00.610ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "371b02cb-56f9-4d42-b1cf-26119d4f3563", "content-length": "52", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
25242025-03-08T00:48:00.610ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
25252025-03-08T00:48:00.610ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
25262025-03-08T00:48:00.628ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
25272025-03-08T00:48:00.740ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8fbd92d2-5358-4dbf-b41e-c51fdfc9ef1d", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
25282025-03-08T00:48:00.740ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
25292025-03-08T00:48:00.740ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
25302025-03-08T00:48:00.740ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "036a872a-88b6-438f-bb60-b43e88073028", "content-length": "691", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
25312025-03-08T00:48:00.740ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
25322025-03-08T00:48:00.740ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
25332025-03-08T00:48:00.740ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
25342025-03-08T00:48:00.740ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "38a4ca02-7b36-4756-8cba-f351a10e9360", "content-length": "682", "date": "Sat, 08 Mar 2025 00:48:00 GMT"} })
25352025-03-08T00:48:00.740ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25362025-03-08T00:48:00.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25372025-03-08T00:48:00.741ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 448.408924ms iteration = 16 status = {}
25382025-03-08T00:48:00.745ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25392025-03-08T00:48:00.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25402025-03-08T00:48:00.748ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25412025-03-08T00:48:00.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25422025-03-08T00:48:00.750ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25432025-03-08T00:48:00.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25442025-03-08T00:48:00.757ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 464.509001ms iteration = 16 status = {}
25452025-03-08T00:48:02.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 329 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = f207953f-25b9-44a1-81c5-0b0eb19fb7c1 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
25462025-03-08T00:48:10.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
25472025-03-08T00:48:10.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25482025-03-08T00:48:10.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25492025-03-08T00:48:10.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
25502025-03-08T00:48:10.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25512025-03-08T00:48:10.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25522025-03-08T00:48:10.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25532025-03-08T00:48:10.295ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25542025-03-08T00:48:10.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25552025-03-08T00:48:10.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25562025-03-08T00:48:10.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25572025-03-08T00:48:10.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25582025-03-08T00:48:10.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25592025-03-08T00:48:10.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25602025-03-08T00:48:10.305ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25612025-03-08T00:48:10.308ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
25622025-03-08T00:48:10.308ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25632025-03-08T00:48:10.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25642025-03-08T00:48:10.320ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
25652025-03-08T00:48:10.320ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.383765ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
25662025-03-08T00:48:10.331ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.757081ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25672025-03-08T00:48:12.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 378 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = a9ee652f-2b0a-4bdc-9961-2d8bdfda2cff response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
25682025-03-08T00:48:20.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
25692025-03-08T00:48:20.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25702025-03-08T00:48:20.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25712025-03-08T00:48:20.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
25722025-03-08T00:48:20.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25732025-03-08T00:48:20.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25742025-03-08T00:48:20.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25752025-03-08T00:48:20.295ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25762025-03-08T00:48:20.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25772025-03-08T00:48:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25782025-03-08T00:48:20.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25792025-03-08T00:48:20.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25802025-03-08T00:48:20.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25812025-03-08T00:48:20.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25822025-03-08T00:48:20.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25832025-03-08T00:48:20.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
25842025-03-08T00:48:20.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25852025-03-08T00:48:20.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25862025-03-08T00:48:20.312ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
25872025-03-08T00:48:20.312ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.486301ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
25882025-03-08T00:48:20.323ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 29.619171ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25892025-03-08T00:48:22.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 364 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 942213f6-a09d-4520-a8ec-c07dead059e9 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
25902025-03-08T00:48:30.292ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
25912025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
25922025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
25932025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
25942025-03-08T00:48:30.293ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-08 00:38:30.292972368 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
25952025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
25962025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
25972025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
25982025-03-08T00:48:30.293ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = a346f089-8033-4b92-b854-361d1da5cab2
25992025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 21.531\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"a346f089-8033-4b92-b854-361d1da5cab2"}
26002025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
26012025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
26022025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26032025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
26042025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26052025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
26062025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
26072025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
26082025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
26092025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26102025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
26112025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
26122025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
26132025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
26142025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
26152025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26162025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
26172025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
26182025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
26192025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
26202025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
26212025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26222025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26232025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
26242025-03-08T00:48:30.293ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
26252025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
26262025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
26272025-03-08T00:48:30.293ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
26282025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
26292025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26302025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26312025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
26322025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26332025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
26342025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
26352025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
26362025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
26372025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26382025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
26392025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
26402025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
26412025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
26422025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26432025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
26442025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
26452025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
26462025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26472025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26482025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
26492025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
26502025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
26512025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26522025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
26532025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
26542025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
26552025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
26562025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
26572025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
26582025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26592025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26602025-03-08T00:48:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
26612025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
26622025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26632025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26642025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
26652025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26662025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
26672025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26682025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26692025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26702025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
26712025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26722025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
26732025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26742025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
26752025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26762025-03-08T00:48:30.294ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1ac98a83-1243-46f5-99db-fd29222026d8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("508de01a-93ec-4309-a59e-0cfd22b5027a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1393, tv_nsec: 317331966 } })
26772025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26782025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26792025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
26802025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26812025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26822025-03-08T00:48:30.294ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("59287b7e-0d61-4586-b7c5-e79476927737.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1393, tv_nsec: 317388238 } })
26832025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
26842025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
26852025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
26862025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.786154ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
26872025-03-08T00:48:30.294ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.700602ms iteration = 9 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
26882025-03-08T00:48:30.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
26892025-03-08T00:48:30.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26902025-03-08T00:48:30.296ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-08 00:38:30.292972368 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
26912025-03-08T00:48:30.296ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.438155ms iteration = 9 status = {"expiration":"2025-03-08T00:38:30.292972368Z","pruned":[]}
26922025-03-08T00:48:30.296ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1393, tv_nsec: 320001402 } })
26932025-03-08T00:48:30.297ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
26942025-03-08T00:48:30.297ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.088431ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
26952025-03-08T00:48:30.297ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1393, tv_nsec: 320388512 } })
26962025-03-08T00:48:30.297ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 4.155773ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
26972025-03-08T00:48:30.297ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
26982025-03-08T00:48:30.298ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26992025-03-08T00:48:30.299ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27002025-03-08T00:48:30.299ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27012025-03-08T00:48:30.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27022025-03-08T00:48:30.300ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
27032025-03-08T00:48:30.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
27042025-03-08T00:48:30.301ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27052025-03-08T00:48:30.301ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27062025-03-08T00:48:30.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27072025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27082025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27092025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
27102025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27112025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
27122025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
27132025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27142025-03-08T00:48:30.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
27152025-03-08T00:48:30.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27162025-03-08T00:48:30.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27172025-03-08T00:48:30.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27182025-03-08T00:48:30.304ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27192025-03-08T00:48:30.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27202025-03-08T00:48:30.305ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27212025-03-08T00:48:30.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27222025-03-08T00:48:30.307ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27232025-03-08T00:48:30.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27242025-03-08T00:48:30.309ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27252025-03-08T00:48:30.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
27262025-03-08T00:48:30.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27272025-03-08T00:48:30.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27282025-03-08T00:48:30.316ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27292025-03-08T00:48:30.319ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27302025-03-08T00:48:30.319ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27312025-03-08T00:48:30.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27322025-03-08T00:48:30.322ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27332025-03-08T00:48:30.324ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27342025-03-08T00:48:30.324ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
27352025-03-08T00:48:30.324ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27362025-03-08T00:48:30.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27372025-03-08T00:48:30.327ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
27382025-03-08T00:48:30.327ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27392025-03-08T00:48:30.328ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
27402025-03-08T00:48:30.328ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
27412025-03-08T00:48:30.328ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
27422025-03-08T00:48:30.328ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27432025-03-08T00:48:30.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27442025-03-08T00:48:30.328ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27452025-03-08T00:48:30.331ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27462025-03-08T00:48:30.331ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27472025-03-08T00:48:30.332ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27482025-03-08T00:48:30.333ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
27492025-03-08T00:48:30.333ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 39.845714ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
27502025-03-08T00:48:30.333ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27512025-03-08T00:48:30.336ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1393, tv_nsec: 359187559 } })
27522025-03-08T00:48:30.336ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27532025-03-08T00:48:30.336ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
27542025-03-08T00:48:30.336ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
27552025-03-08T00:48:30.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27562025-03-08T00:48:30.336ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
27572025-03-08T00:48:30.336ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
27582025-03-08T00:48:30.336ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27592025-03-08T00:48:30.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27602025-03-08T00:48:30.338ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27612025-03-08T00:48:30.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27622025-03-08T00:48:30.338ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27632025-03-08T00:48:30.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27642025-03-08T00:48:30.342ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27652025-03-08T00:48:30.345ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 52.230129ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
27662025-03-08T00:48:30.347ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27672025-03-08T00:48:30.347ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27682025-03-08T00:48:30.347ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27692025-03-08T00:48:30.352ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27702025-03-08T00:48:30.356ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
27712025-03-08T00:48:30.356ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
27722025-03-08T00:48:30.356ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27732025-03-08T00:48:30.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27742025-03-08T00:48:30.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
27752025-03-08T00:48:30.361ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 67.655909ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
27762025-03-08T00:48:30.362ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27772025-03-08T00:48:30.365ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
27782025-03-08T00:48:30.365ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27792025-03-08T00:48:30.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27802025-03-08T00:48:30.365ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
27812025-03-08T00:48:30.365ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
27822025-03-08T00:48:30.365ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 72.496469ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
27832025-03-08T00:48:30.368ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27842025-03-08T00:48:30.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27852025-03-08T00:48:30.369ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 76.234161ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
27862025-03-08T00:48:30.383ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
27872025-03-08T00:48:30.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27882025-03-08T00:48:30.395ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27892025-03-08T00:48:30.396ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27902025-03-08T00:48:30.397ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27912025-03-08T00:48:30.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27922025-03-08T00:48:30.398ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
27932025-03-08T00:48:30.398ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
27942025-03-08T00:48:30.398ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 105.248887ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
27952025-03-08T00:48:30.399ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27962025-03-08T00:48:30.399ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
27972025-03-08T00:48:30.399ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27982025-03-08T00:48:30.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27992025-03-08T00:48:30.400ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
28002025-03-08T00:48:30.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28012025-03-08T00:48:30.401ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
28022025-03-08T00:48:30.401ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28032025-03-08T00:48:30.401ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
28042025-03-08T00:48:30.401ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28052025-03-08T00:48:30.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28062025-03-08T00:48:30.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28072025-03-08T00:48:30.402ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 109.063301ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
28082025-03-08T00:48:30.438ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
28092025-03-08T00:48:30.443ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
28102025-03-08T00:48:30.443ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28112025-03-08T00:48:30.443ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad73a9f7-3ec0-4896-98cc-8e92cc9accbc", "content-length": "26", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
28122025-03-08T00:48:30.443ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28132025-03-08T00:48:30.446ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28142025-03-08T00:48:30.448ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
28152025-03-08T00:48:30.449ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
28162025-03-08T00:48:30.449ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28172025-03-08T00:48:30.449ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dd06add-a5d7-429d-a251-102c49e88ce0", "content-length": "26", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
28182025-03-08T00:48:30.449ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28192025-03-08T00:48:30.451ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28202025-03-08T00:48:30.494ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
28212025-03-08T00:48:30.494ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28222025-03-08T00:48:30.494ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d0d3c53-bbbc-405e-9b95-3b24780bc384", "content-length": "26", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
28232025-03-08T00:48:30.494ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28242025-03-08T00:48:30.496ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
28252025-03-08T00:48:30.496ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
28262025-03-08T00:48:30.496ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
28272025-03-08T00:48:30.497ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e92caf60-14b3-4abe-a9ea-a3c0ccd62a1d", "content-length": "372", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
28282025-03-08T00:48:30.497ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28292025-03-08T00:48:30.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28302025-03-08T00:48:30.499ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28312025-03-08T00:48:30.500ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28322025-03-08T00:48:30.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28332025-03-08T00:48:30.501ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28342025-03-08T00:48:30.501ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28352025-03-08T00:48:30.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2836 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
28372025-03-08T00:48:30.508ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
28382025-03-08T00:48:30.508ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 214.624535ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
28392025-03-08T00:48:30.510ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28402025-03-08T00:48:30.515ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
28412025-03-08T00:48:30.515ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28422025-03-08T00:48:30.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28432025-03-08T00:48:30.518ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
28442025-03-08T00:48:30.518ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 224.982231ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
28452025-03-08T00:48:30.526ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28462025-03-08T00:48:30.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28472025-03-08T00:48:30.529ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28482025-03-08T00:48:30.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28492025-03-08T00:48:30.531ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28502025-03-08T00:48:30.534ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 241.332675ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28512025-03-08T00:48:30.534ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 241.137099ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28522025-03-08T00:48:30.535ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
28532025-03-08T00:48:30.535ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28542025-03-08T00:48:30.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28552025-03-08T00:48:30.535ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
28562025-03-08T00:48:30.535ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
28572025-03-08T00:48:30.535ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28582025-03-08T00:48:30.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28592025-03-08T00:48:30.537ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28602025-03-08T00:48:30.537ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 244.466472ms iteration = 17 status = 0
28612025-03-08T00:48:30.540ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28622025-03-08T00:48:30.540ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
28632025-03-08T00:48:30.540ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 246.83769ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
28642025-03-08T00:48:30.540ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
28652025-03-08T00:48:30.540ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28662025-03-08T00:48:30.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28672025-03-08T00:48:30.543ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
28682025-03-08T00:48:30.543ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
28692025-03-08T00:48:30.543ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-08 00:40:30.694594 UTC dns_group = internal generation = 1
28702025-03-08T00:48:30.543ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 249.972968ms iteration = 9 status = {"generation":1}
28712025-03-08T00:48:30.547ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
28722025-03-08T00:48:30.547ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 0 version = 2
28732025-03-08T00:48:30.547ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28742025-03-08T00:48:30.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28752025-03-08T00:48:30.550ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28762025-03-08T00:48:30.551ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28772025-03-08T00:48:30.553ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
28782025-03-08T00:48:30.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
28792025-03-08T00:48:30.554ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
28802025-03-08T00:48:30.554ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
28812025-03-08T00:48:30.554ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28822025-03-08T00:48:30.555ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
28832025-03-08T00:48:30.556ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28842025-03-08T00:48:30.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28852025-03-08T00:48:30.558ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 1 version = 2
28862025-03-08T00:48:30.558ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
28872025-03-08T00:48:30.558ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-08 00:40:32.180226 UTC dns_group = external generation = 2
28882025-03-08T00:48:30.558ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 265.773548ms iteration = 9 status = {"generation":2}
28892025-03-08T00:48:30.565ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
28902025-03-08T00:48:30.565ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
28912025-03-08T00:48:30.566ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28922025-03-08T00:48:30.566ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28932025-03-08T00:48:30.566ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a9d669aa-d66a-43d3-a805-dcddd40abff8", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
28942025-03-08T00:48:30.566ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28952025-03-08T00:48:30.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28962025-03-08T00:48:30.568ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28972025-03-08T00:48:30.570ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28982025-03-08T00:48:30.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28992025-03-08T00:48:30.571ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
29002025-03-08T00:48:30.571ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29012025-03-08T00:48:30.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
29022025-03-08T00:48:30.574ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29032025-03-08T00:48:30.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29042025-03-08T00:48:30.581ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-08T00:40:26.976319Z, time_modified: 2025-03-08T00:40:26.976319Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }) }
29052025-03-08T00:48:30.582ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 288.757865ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]}},"default_endpoint":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
29062025-03-08T00:48:30.583ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
29072025-03-08T00:48:30.583ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "71529224-970c-4c51-8f09-b0ad199811a3", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29082025-03-08T00:48:30.583ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29092025-03-08T00:48:30.585ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "f2f718d0-2214-4f64-b55d-9aea6c028037", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29102025-03-08T00:48:30.585ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29112025-03-08T00:48:30.586ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
29122025-03-08T00:48:30.586ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29132025-03-08T00:48:30.586ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "86c02ed5-5eb0-42da-acaa-204c7c8f39f9", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
29142025-03-08T00:48:30.589ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29152025-03-08T00:48:30.589ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 296.524757ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
29162025-03-08T00:48:30.591ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 297.929412ms iteration = 17 status = {}
29172025-03-08T00:48:30.592ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
29182025-03-08T00:48:30.597ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
29192025-03-08T00:48:30.597ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "232f4d03-0e29-4793-920e-5831121677cf", "content-length": "1", "date": "Sat, 08 Mar 2025 00:48:29 GMT"} })
29202025-03-08T00:48:30.599ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
29212025-03-08T00:48:30.599ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
29222025-03-08T00:48:30.599ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
29232025-03-08T00:48:30.599ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 306.296787ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
29242025-03-08T00:48:30.601ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 307.755074ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
29252025-03-08T00:48:30.601ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 308.632235ms iteration = 17 status = null
29262025-03-08T00:48:30.604ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 311.26727ms iteration = 17 status = {}
29272025-03-08T00:48:30.636ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29282025-03-08T00:48:30.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29292025-03-08T00:48:30.638ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29302025-03-08T00:48:30.641ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
29312025-03-08T00:48:30.641ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29322025-03-08T00:48:30.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29332025-03-08T00:48:30.642ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 348.54002ms iteration = 18 status = {"physical_disks_added":0}
29342025-03-08T00:48:30.645ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
29352025-03-08T00:48:30.645ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "47418ee8-3ca3-405b-b40f-8617466ced0d", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
29362025-03-08T00:48:30.645ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
29372025-03-08T00:48:30.648ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "c77b9329-b1cd-40fe-84ff-d11addf65d75", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
29382025-03-08T00:48:30.648ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
29392025-03-08T00:48:30.650ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "5ea61c54-d8ec-40f0-9a42-34d0d4fce464", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
29402025-03-08T00:48:30.650ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
29412025-03-08T00:48:30.652ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
29422025-03-08T00:48:30.652ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 359.830779ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
29432025-03-08T00:48:30.697ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29442025-03-08T00:48:30.700ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
29452025-03-08T00:48:30.700ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29462025-03-08T00:48:30.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29472025-03-08T00:48:30.707ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29482025-03-08T00:48:30.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29492025-03-08T00:48:30.710ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
29502025-03-08T00:48:30.710ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29512025-03-08T00:48:30.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29522025-03-08T00:48:30.762ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
29532025-03-08T00:48:30.762ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e64dfe89-8a3c-43ea-a4f5-1e3b0b3b0d41", "content-length": "691", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
29542025-03-08T00:48:30.762ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
29552025-03-08T00:48:30.762ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
29562025-03-08T00:48:30.762ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
29572025-03-08T00:48:30.764ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 470.761967ms iteration = 17 status = {}
29582025-03-08T00:48:30.772ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
29592025-03-08T00:48:30.773ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "771cbd79-36c6-477e-8df4-9ef2a979a68c", "content-length": "134", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
29602025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
29612025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
29622025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
29632025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
29642025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29652025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29662025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29672025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
29682025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29692025-03-08T00:48:30.773ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
29702025-03-08T00:48:30.774ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
29712025-03-08T00:48:30.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29722025-03-08T00:48:30.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29732025-03-08T00:48:30.777ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
29742025-03-08T00:48:30.777ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ef2782-bf7d-43d6-b0ef-59f72a3742e4", "content-length": "2", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
29752025-03-08T00:48:30.777ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
29762025-03-08T00:48:30.777ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0005a8a-df48-4dcf-82a0-7026a668ca3d", "content-length": "52", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
29772025-03-08T00:48:30.777ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
29782025-03-08T00:48:30.777ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
29792025-03-08T00:48:30.875ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "da65ba8e-d14a-42e7-be3c-89789b2014b3", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
29802025-03-08T00:48:30.875ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
29812025-03-08T00:48:30.875ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
29822025-03-08T00:48:30.875ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaefec57-3342-440e-88b4-c7c786262de1", "content-length": "682", "date": "Sat, 08 Mar 2025 00:48:30 GMT"} })
29832025-03-08T00:48:30.875ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29842025-03-08T00:48:30.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29852025-03-08T00:48:30.880ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29862025-03-08T00:48:30.880ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29872025-03-08T00:48:30.884ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29882025-03-08T00:48:30.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29892025-03-08T00:48:30.886ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29902025-03-08T00:48:30.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29912025-03-08T00:48:30.893ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 600.446366ms iteration = 17 status = {}
29922025-03-08T00:48:32.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 340 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = a677bf84-5deb-4fbf-9953-9025c772cc80 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
29932025-03-08T00:48:40.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
29942025-03-08T00:48:40.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29952025-03-08T00:48:40.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29962025-03-08T00:48:40.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
29972025-03-08T00:48:40.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29982025-03-08T00:48:40.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29992025-03-08T00:48:40.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30002025-03-08T00:48:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30012025-03-08T00:48:40.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30022025-03-08T00:48:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30032025-03-08T00:48:40.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30042025-03-08T00:48:40.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30052025-03-08T00:48:40.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30062025-03-08T00:48:40.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
30072025-03-08T00:48:40.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30082025-03-08T00:48:40.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30092025-03-08T00:48:40.315ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
30102025-03-08T00:48:40.315ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.318525ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
30112025-03-08T00:48:40.324ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30122025-03-08T00:48:40.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30132025-03-08T00:48:40.325ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.579644ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
30142025-03-08T00:48:42.640ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:43106
30152025-03-08T00:48:42.640ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = 85497811-f3b9-4a31-a224-a7b35e7eef4d roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
30162025-03-08T00:48:42.640ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = 85497811-f3b9-4a31-a224-a7b35e7eef4d resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
30172025-03-08T00:48:42.643ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2936 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = 85497811-f3b9-4a31-a224-a7b35e7eef4d response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
30182025-03-08T00:48:42.643ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = 7f1d6021-f3a3-4ad2-affc-e34863163858 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
30192025-03-08T00:48:42.643ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = 7f1d6021-f3a3-4ad2-affc-e34863163858 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
30202025-03-08T00:48:42.645ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2323 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = 7f1d6021-f3a3-4ad2-affc-e34863163858 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
30212025-03-08T00:48:42.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 334 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = f332ca62-e7a3-46ff-9969-721861c81b94 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
30222025-03-08T00:48:50.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
30232025-03-08T00:48:50.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30242025-03-08T00:48:50.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30252025-03-08T00:48:50.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30262025-03-08T00:48:50.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
30272025-03-08T00:48:50.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30282025-03-08T00:48:50.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30292025-03-08T00:48:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30302025-03-08T00:48:50.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30312025-03-08T00:48:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30322025-03-08T00:48:50.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30332025-03-08T00:48:50.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30342025-03-08T00:48:50.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30352025-03-08T00:48:50.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
30362025-03-08T00:48:50.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30372025-03-08T00:48:50.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30382025-03-08T00:48:50.316ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
30392025-03-08T00:48:50.316ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.70432ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
30402025-03-08T00:48:50.323ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30412025-03-08T00:48:50.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30422025-03-08T00:48:50.353ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.573521ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
30432025-03-08T00:48:52.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 369 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 177981c6-e72f-40da-8a90-c27447ef297d response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
30442025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
30452025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
30462025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
30472025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
30482025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
30492025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
30502025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
30512025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
30522025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30532025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
30542025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
30552025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
30562025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
30572025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30582025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30592025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
30602025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
30612025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
30622025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
30632025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
30642025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
30652025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
30662025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
30672025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
30682025-03-08T00:49:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
30692025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
30702025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
30712025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
30722025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30732025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
30742025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
30752025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
30762025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
30772025-03-08T00:49:00.293ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
30782025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30792025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
30802025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
30812025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
30822025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30832025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
30842025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
30852025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
30862025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30872025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
30882025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
30892025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
30902025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
30912025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30922025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
30932025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
30942025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
30952025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30962025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
30972025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30982025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30992025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31002025-03-08T00:49:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
31012025-03-08T00:49:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1423, tv_nsec: 318699980 } })
31022025-03-08T00:49:00.295ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
31032025-03-08T00:49:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1423, tv_nsec: 319237683 } })
31042025-03-08T00:49:00.296ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.446775ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
31052025-03-08T00:49:00.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
31062025-03-08T00:49:00.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31072025-03-08T00:49:00.296ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31082025-03-08T00:49:00.297ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31092025-03-08T00:49:00.297ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31102025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
31112025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
31122025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
31132025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31142025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
31152025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31162025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
31172025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31182025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
31192025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
31202025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31212025-03-08T00:49:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31222025-03-08T00:49:00.302ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 9.749381ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
31232025-03-08T00:49:00.303ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
31242025-03-08T00:49:00.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
31252025-03-08T00:49:00.304ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31262025-03-08T00:49:00.305ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31272025-03-08T00:49:00.307ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
31282025-03-08T00:49:00.307ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31292025-03-08T00:49:00.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31302025-03-08T00:49:00.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
31312025-03-08T00:49:00.309ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31322025-03-08T00:49:00.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31332025-03-08T00:49:00.311ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31342025-03-08T00:49:00.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31352025-03-08T00:49:00.312ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
31362025-03-08T00:49:00.312ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31372025-03-08T00:49:00.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31382025-03-08T00:49:00.313ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
31392025-03-08T00:49:00.313ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 20.424594ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
31402025-03-08T00:49:00.314ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1423, tv_nsec: 337843353 } })
31412025-03-08T00:49:00.314ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
31422025-03-08T00:49:00.315ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
31432025-03-08T00:49:00.315ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31442025-03-08T00:49:00.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31452025-03-08T00:49:00.354ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31462025-03-08T00:49:00.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31472025-03-08T00:49:00.356ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 63.097577ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
31482025-03-08T00:49:00.357ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31492025-03-08T00:49:00.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31502025-03-08T00:49:00.358ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 65.974528ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
31512025-03-08T00:49:00.360ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31522025-03-08T00:49:00.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
31532025-03-08T00:49:00.364ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31542025-03-08T00:49:00.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31552025-03-08T00:49:00.371ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
31562025-03-08T00:49:00.371ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31572025-03-08T00:49:00.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31582025-03-08T00:49:00.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
31592025-03-08T00:49:00.372ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31602025-03-08T00:49:00.375ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
31612025-03-08T00:49:00.375ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31622025-03-08T00:49:00.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31632025-03-08T00:49:00.376ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
31642025-03-08T00:49:00.376ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31652025-03-08T00:49:00.376ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
31662025-03-08T00:49:00.376ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
31672025-03-08T00:49:00.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31682025-03-08T00:49:00.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31692025-03-08T00:49:00.379ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
31702025-03-08T00:49:00.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
31712025-03-08T00:49:00.379ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31722025-03-08T00:49:00.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31732025-03-08T00:49:00.381ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
31742025-03-08T00:49:00.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
31752025-03-08T00:49:00.381ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 88.418021ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
31762025-03-08T00:49:00.383ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 90.630526ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
31772025-03-08T00:49:00.385ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31782025-03-08T00:49:00.385ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
31792025-03-08T00:49:00.385ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31802025-03-08T00:49:00.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
3181 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
31822025-03-08T00:49:00.388ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
31832025-03-08T00:49:00.388ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
31842025-03-08T00:49:00.388ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31852025-03-08T00:49:00.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
31862025-03-08T00:49:00.402ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 109.945423ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31872025-03-08T00:49:00.404ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
31882025-03-08T00:49:00.404ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 112.014633ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
31892025-03-08T00:49:00.412ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31902025-03-08T00:49:00.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31912025-03-08T00:49:00.415ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31922025-03-08T00:49:00.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31932025-03-08T00:49:00.417ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
31942025-03-08T00:49:00.417ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 124.517443ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
31952025-03-08T00:49:00.417ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31962025-03-08T00:49:00.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
31972025-03-08T00:49:00.420ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31982025-03-08T00:49:00.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31992025-03-08T00:49:00.422ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 129.895055ms iteration = 18 status = 0
32002025-03-08T00:49:00.424ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
32012025-03-08T00:49:00.424ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32022025-03-08T00:49:00.424ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1628d61b-a241-44c7-81a0-644e785ebeff", "content-length": "26", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32032025-03-08T00:49:00.424ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32042025-03-08T00:49:00.426ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
32052025-03-08T00:49:00.433ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 136.328884ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
32062025-03-08T00:49:00.433ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
32072025-03-08T00:49:00.433ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
32082025-03-08T00:49:00.433ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 136.881368ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
32092025-03-08T00:49:00.451ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32102025-03-08T00:49:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32112025-03-08T00:49:00.492ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 199.716248ms iteration = 19 status = {"physical_disks_added":0}
32122025-03-08T00:49:00.497ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
32132025-03-08T00:49:00.497ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32142025-03-08T00:49:00.497ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e6470f9-47af-4284-80a8-212f2aec710a", "content-length": "26", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32152025-03-08T00:49:00.497ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32162025-03-08T00:49:00.499ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
32172025-03-08T00:49:00.551ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
32182025-03-08T00:49:00.551ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32192025-03-08T00:49:00.551ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1413ecdb-279b-4fdf-96b5-935eef144700", "content-length": "26", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32202025-03-08T00:49:00.551ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32212025-03-08T00:49:00.553ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
32222025-03-08T00:49:00.576ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
32232025-03-08T00:49:00.576ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
32242025-03-08T00:49:00.576ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3019ac51-0b12-472b-ba39-d1a2cd82f3f9", "content-length": "372", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32252025-03-08T00:49:00.576ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32262025-03-08T00:49:00.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32272025-03-08T00:49:00.619ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
32282025-03-08T00:49:00.620ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "b65522a6-3f06-4f4b-b2b1-f72d111de92d", "content-length": "1", "date": "Sat, 08 Mar 2025 00:48:59 GMT"} })
32292025-03-08T00:49:00.621ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
32302025-03-08T00:49:00.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
32312025-03-08T00:49:00.624ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 331.635243ms iteration = 18 status = null
32322025-03-08T00:49:00.627ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
32332025-03-08T00:49:00.627ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
32342025-03-08T00:49:00.657ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 364.612537ms iteration = 18 status = {}
32352025-03-08T00:49:00.700ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32362025-03-08T00:49:00.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32372025-03-08T00:49:00.702ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32382025-03-08T00:49:00.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32392025-03-08T00:49:00.703ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
32402025-03-08T00:49:00.703ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32412025-03-08T00:49:00.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32422025-03-08T00:49:00.714ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
32432025-03-08T00:49:00.714ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "062c78ad-00e6-47ab-98c9-8eded86a40f8", "content-length": "134", "date": "Sat, 08 Mar 2025 00:48:59 GMT"} })
32442025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
32452025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
32462025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
32472025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
32482025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32492025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32502025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32512025-03-08T00:49:00.714ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32522025-03-08T00:49:00.715ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
32532025-03-08T00:49:00.715ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
32542025-03-08T00:49:00.715ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
32552025-03-08T00:49:00.715ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32562025-03-08T00:49:00.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32572025-03-08T00:49:00.718ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
32582025-03-08T00:49:00.718ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "0641edae-89f2-4de3-8d4d-04a3d8b30116", "content-length": "2", "date": "Sat, 08 Mar 2025 00:48:59 GMT"} })
32592025-03-08T00:49:00.718ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
32602025-03-08T00:49:00.718ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "653ded1c-8a92-41fc-a6c6-7107297088e6", "content-length": "52", "date": "Sat, 08 Mar 2025 00:48:59 GMT"} })
32612025-03-08T00:49:00.718ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
32622025-03-08T00:49:00.718ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
32632025-03-08T00:49:00.736ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c939b1ca-76e1-4a5b-a8b0-e5e07b265bd3", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32642025-03-08T00:49:00.736ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32652025-03-08T00:49:00.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32662025-03-08T00:49:00.738ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32672025-03-08T00:49:00.742ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32682025-03-08T00:49:00.742ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32692025-03-08T00:49:00.742ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32702025-03-08T00:49:00.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32712025-03-08T00:49:00.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32722025-03-08T00:49:00.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32732025-03-08T00:49:00.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32742025-03-08T00:49:00.751ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
32752025-03-08T00:49:00.751ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
32762025-03-08T00:49:00.751ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "9b80f369-e243-4288-9861-c553e406ce44", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32772025-03-08T00:49:00.753ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 459.993081ms iteration = 18 status = {}
32782025-03-08T00:49:00.773ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32792025-03-08T00:49:00.775ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32802025-03-08T00:49:00.775ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32812025-03-08T00:49:00.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32822025-03-08T00:49:00.851ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4eda98ec-7460-4b9c-b921-6e2c774c9f4f", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32832025-03-08T00:49:00.851ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
32842025-03-08T00:49:00.851ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
32852025-03-08T00:49:00.851ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "87cad39b-2421-4185-bdf2-bb4feea81b9a", "content-length": "682", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32862025-03-08T00:49:00.851ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32872025-03-08T00:49:00.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32882025-03-08T00:49:00.855ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32892025-03-08T00:49:00.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32902025-03-08T00:49:00.859ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32912025-03-08T00:49:00.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32922025-03-08T00:49:00.879ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
32932025-03-08T00:49:00.879ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "61b562c7-4579-43eb-8671-ec5de10a1c2d", "content-length": "691", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
32942025-03-08T00:49:00.879ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
32952025-03-08T00:49:00.879ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
32962025-03-08T00:49:00.879ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
32972025-03-08T00:49:00.879ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 587.140847ms iteration = 18 status = {}
32982025-03-08T00:49:00.881ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32992025-03-08T00:49:00.881ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33002025-03-08T00:49:00.888ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 595.725019ms iteration = 18 status = {}
33012025-03-08T00:49:00.945ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
33022025-03-08T00:49:00.945ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
33032025-03-08T00:49:00.985ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
33042025-03-08T00:49:00.985ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:36141
33052025-03-08T00:49:00.985ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:36141 req_id = 24ea4dae-d5cd-40f1-920f-5fa81cb126e9 roles = RoleSet { roles: {} } uri = /metrics/producers
33062025-03-08T00:49:00.985ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:36141 req_id = 24ea4dae-d5cd-40f1-920f-5fa81cb126e9 resource = Database result = Ok(()) uri = /metrics/producers
33072025-03-08T00:49:01.036ZINFOnexus: assigned collector to new producer collector_id = c5a56912-50aa-493d-9612-f426cf7f1072 file = nexus/src/app/oximeter.rs:76 producer_id = 841ee309-4e1e-443f-ae06-2ad5eb454a8d
33082025-03-08T00:49:01.036ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 51165 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:36141 req_id = 24ea4dae-d5cd-40f1-920f-5fa81cb126e9 response_code = 201 uri = /metrics/producers
33092025-03-08T00:49:01.036ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "24ea4dae-d5cd-40f1-920f-5fa81cb126e9", "content-length": "41", "date": "Sat, 08 Mar 2025 00:49:00 GMT"} })
33102025-03-08T00:49:01.038ZDEBGnexus (producer-registration-task): registered with nexus successfully
33112025-03-08T00:49:01.038ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
33122025-03-08T00:49:02.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 380 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = cb805fdf-9be5-450e-8500-2630bb069754 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
33132025-03-08T00:49:10.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
33142025-03-08T00:49:10.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33152025-03-08T00:49:10.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33162025-03-08T00:49:10.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
33172025-03-08T00:49:10.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33182025-03-08T00:49:10.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33192025-03-08T00:49:10.325ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33202025-03-08T00:49:10.328ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33212025-03-08T00:49:10.328ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33222025-03-08T00:49:10.328ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33232025-03-08T00:49:10.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33242025-03-08T00:49:10.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33252025-03-08T00:49:10.330ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33262025-03-08T00:49:10.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33272025-03-08T00:49:10.331ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33282025-03-08T00:49:10.334ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
33292025-03-08T00:49:10.334ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33302025-03-08T00:49:10.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33312025-03-08T00:49:10.347ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
33322025-03-08T00:49:10.347ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 54.921305ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
33332025-03-08T00:49:10.359ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.079851ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33342025-03-08T00:49:12.639ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = db6c9fb8-1744-4e66-8452-8dfd02ce5fc8 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
33352025-03-08T00:49:12.640ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = db6c9fb8-1744-4e66-8452-8dfd02ce5fc8 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
33362025-03-08T00:49:12.642ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2859 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = db6c9fb8-1744-4e66-8452-8dfd02ce5fc8 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
33372025-03-08T00:49:12.643ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = be9684c3-e670-4a69-a9f7-2d4fbd05dda8 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
33382025-03-08T00:49:12.643ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = be9684c3-e670-4a69-a9f7-2d4fbd05dda8 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
33392025-03-08T00:49:12.645ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2172 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43106 req_id = be9684c3-e670-4a69-a9f7-2d4fbd05dda8 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
33402025-03-08T00:49:12.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 294 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = ee3e919d-40a3-4025-befa-7052c696b96a response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
33412025-03-08T00:49:20.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
33422025-03-08T00:49:20.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33432025-03-08T00:49:20.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33442025-03-08T00:49:20.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
33452025-03-08T00:49:20.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33462025-03-08T00:49:20.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33472025-03-08T00:49:20.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33482025-03-08T00:49:20.295ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33492025-03-08T00:49:20.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33502025-03-08T00:49:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
33512025-03-08T00:49:20.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33522025-03-08T00:49:20.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33532025-03-08T00:49:20.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33542025-03-08T00:49:20.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33552025-03-08T00:49:20.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33562025-03-08T00:49:20.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
33572025-03-08T00:49:20.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33582025-03-08T00:49:20.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33592025-03-08T00:49:20.316ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
33602025-03-08T00:49:20.316ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.490874ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
33612025-03-08T00:49:20.325ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.176029ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33622025-03-08T00:49:22.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 331 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 87b38435-0e5b-48f6-b49c-c6906384467a response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
33632025-03-08T00:49:30.291ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
33642025-03-08T00:49:30.291ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
33652025-03-08T00:49:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
33662025-03-08T00:49:30.291ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
33672025-03-08T00:49:30.291ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-08 00:39:30.291880229 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
33682025-03-08T00:49:30.291ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
33692025-03-08T00:49:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
33702025-03-08T00:49:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
33712025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
33722025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
33732025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
33742025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
33752025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
33762025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
33772025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
33782025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
33792025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
33802025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
33812025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
33822025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
33832025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
33842025-03-08T00:49:30.292ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = a346f089-8033-4b92-b854-361d1da5cab2
33852025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 15.441\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"a346f089-8033-4b92-b854-361d1da5cab2"}
33862025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
33872025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33882025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
33892025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
33902025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
33912025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
33922025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
33932025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33942025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
33952025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
33962025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
33972025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33982025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
33992025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
34002025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
34012025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
34022025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
34032025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
34042025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34052025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34062025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
34072025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
34082025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
34092025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
34102025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
34112025-03-08T00:49:30.292ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
34122025-03-08T00:49:30.293ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
34132025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
34142025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34152025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
34162025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34172025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
34182025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
34192025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
34202025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
34212025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
34222025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34232025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
34242025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34252025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
34262025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
34272025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
34282025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
34292025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
34302025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
34312025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
34322025-03-08T00:49:30.293ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
34332025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
34342025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
34352025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34362025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
34372025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
34382025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
34392025-03-08T00:49:30.293ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("59287b7e-0d61-4586-b7c5-e79476927737.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1453, tv_nsec: 316698601 } })
34402025-03-08T00:49:30.293ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1ac98a83-1243-46f5-99db-fd29222026d8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("508de01a-93ec-4309-a59e-0cfd22b5027a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1453, tv_nsec: 316700451 } })
34412025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34422025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34432025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
34442025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
34452025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
34462025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
34472025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
34482025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
34492025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34502025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
34512025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
34522025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34532025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34542025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34552025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
34562025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
34572025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
34582025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
34592025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
34602025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34612025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
34622025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.783204ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
34632025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
34642025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
34652025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
34662025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.874166ms iteration = 10 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
34672025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
34682025-03-08T00:49:30.293ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
34692025-03-08T00:49:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
34702025-03-08T00:49:30.294ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
34712025-03-08T00:49:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
34722025-03-08T00:49:30.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
34732025-03-08T00:49:30.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
34742025-03-08T00:49:30.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1453, tv_nsec: 319131011 } })
34752025-03-08T00:49:30.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1453, tv_nsec: 319216903 } })
34762025-03-08T00:49:30.295ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
34772025-03-08T00:49:30.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
34782025-03-08T00:49:30.296ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-08 00:39:30.291880229 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
34792025-03-08T00:49:30.296ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.578923ms iteration = 10 status = {"expiration":"2025-03-08T00:39:30.291880229Z","pruned":[]}
34802025-03-08T00:49:30.296ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
34812025-03-08T00:49:30.296ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.781988ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
34822025-03-08T00:49:30.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34832025-03-08T00:49:30.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
34842025-03-08T00:49:30.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34852025-03-08T00:49:30.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34862025-03-08T00:49:30.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34872025-03-08T00:49:30.303ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.855198ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
34882025-03-08T00:49:30.305ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
34892025-03-08T00:49:30.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
34902025-03-08T00:49:30.309ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34912025-03-08T00:49:30.312ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
34922025-03-08T00:49:30.312ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
34932025-03-08T00:49:30.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
34942025-03-08T00:49:30.313ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34952025-03-08T00:49:30.315ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34962025-03-08T00:49:30.315ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34972025-03-08T00:49:30.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34982025-03-08T00:49:30.323ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34992025-03-08T00:49:30.326ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
35002025-03-08T00:49:30.326ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35012025-03-08T00:49:30.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35022025-03-08T00:49:30.336ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35032025-03-08T00:49:30.342ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
35042025-03-08T00:49:30.342ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35052025-03-08T00:49:30.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35062025-03-08T00:49:30.344ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35072025-03-08T00:49:30.348ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
35082025-03-08T00:49:30.348ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35092025-03-08T00:49:30.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35102025-03-08T00:49:30.350ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35112025-03-08T00:49:30.352ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
35122025-03-08T00:49:30.352ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35132025-03-08T00:49:30.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35142025-03-08T00:49:30.354ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35152025-03-08T00:49:30.355ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35162025-03-08T00:49:30.359ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
35172025-03-08T00:49:30.359ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35182025-03-08T00:49:30.359ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35192025-03-08T00:49:30.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35202025-03-08T00:49:30.359ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
35212025-03-08T00:49:30.360ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 67.356862ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
35222025-03-08T00:49:30.361ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
35232025-03-08T00:49:30.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
35242025-03-08T00:49:30.361ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35252025-03-08T00:49:30.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35262025-03-08T00:49:30.362ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
35272025-03-08T00:49:30.362ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
35282025-03-08T00:49:30.362ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 70.012568ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
35292025-03-08T00:49:30.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
35302025-03-08T00:49:30.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35312025-03-08T00:49:30.365ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35322025-03-08T00:49:30.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35332025-03-08T00:49:30.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35342025-03-08T00:49:30.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35352025-03-08T00:49:30.366ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35362025-03-08T00:49:30.368ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
35372025-03-08T00:49:30.368ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 75.768119ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
35382025-03-08T00:49:30.368ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
35392025-03-08T00:49:30.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
35402025-03-08T00:49:30.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
35412025-03-08T00:49:30.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
35422025-03-08T00:49:30.370ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35432025-03-08T00:49:30.370ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
35442025-03-08T00:49:30.370ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
35452025-03-08T00:49:30.370ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
35462025-03-08T00:49:30.371ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35472025-03-08T00:49:30.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35482025-03-08T00:49:30.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
35492025-03-08T00:49:30.372ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35502025-03-08T00:49:30.372ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
35512025-03-08T00:49:30.372ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
35522025-03-08T00:49:30.372ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 79.83957ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
35532025-03-08T00:49:30.373ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 80.585319ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
35542025-03-08T00:49:30.373ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
35552025-03-08T00:49:30.373ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35562025-03-08T00:49:30.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35572025-03-08T00:49:30.374ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35582025-03-08T00:49:30.374ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35592025-03-08T00:49:30.375ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
35602025-03-08T00:49:30.375ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35612025-03-08T00:49:30.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35622025-03-08T00:49:30.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
35632025-03-08T00:49:30.379ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35642025-03-08T00:49:30.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35652025-03-08T00:49:30.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
35662025-03-08T00:49:30.379ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35672025-03-08T00:49:30.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35682025-03-08T00:49:30.383ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
35692025-03-08T00:49:30.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
35702025-03-08T00:49:30.385ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35712025-03-08T00:49:30.385ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35722025-03-08T00:49:30.389ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
35732025-03-08T00:49:30.389ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35742025-03-08T00:49:30.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35752025-03-08T00:49:30.391ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
35762025-03-08T00:49:30.391ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35772025-03-08T00:49:30.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35782025-03-08T00:49:30.396ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
35792025-03-08T00:49:30.396ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 103.215547ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
35802025-03-08T00:49:30.398ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35812025-03-08T00:49:30.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35822025-03-08T00:49:30.400ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 107.587074ms iteration = 5 status = {"nsuccess":5}
35832025-03-08T00:49:30.400ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35842025-03-08T00:49:30.402ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 109.956013ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
35852025-03-08T00:49:30.403ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35862025-03-08T00:49:30.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35872025-03-08T00:49:30.404ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
35882025-03-08T00:49:30.404ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
35892025-03-08T00:49:30.404ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35902025-03-08T00:49:30.407ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1453, tv_nsec: 431326099 } })
35912025-03-08T00:49:30.408ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
35922025-03-08T00:49:30.408ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35932025-03-08T00:49:30.411ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
35942025-03-08T00:49:30.411ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35952025-03-08T00:49:30.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3596 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
35972025-03-08T00:49:30.423ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
35982025-03-08T00:49:30.423ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
35992025-03-08T00:49:30.423ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
36002025-03-08T00:49:30.423ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36012025-03-08T00:49:30.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36022025-03-08T00:49:30.427ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36032025-03-08T00:49:30.430ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
36042025-03-08T00:49:30.430ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
36052025-03-08T00:49:30.430ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
36062025-03-08T00:49:30.430ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36072025-03-08T00:49:30.430ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36082025-03-08T00:49:30.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36092025-03-08T00:49:30.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
36102025-03-08T00:49:30.431ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
36112025-03-08T00:49:30.431ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
36122025-03-08T00:49:30.431ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
36132025-03-08T00:49:30.431ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 138.446116ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
36142025-03-08T00:49:30.431ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36152025-03-08T00:49:30.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36162025-03-08T00:49:30.432ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
36172025-03-08T00:49:30.432ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 140.459306ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
36182025-03-08T00:49:30.433ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36192025-03-08T00:49:30.433ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36202025-03-08T00:49:30.437ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 145.180023ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36212025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
36222025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36232025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36242025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
36252025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36262025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
36272025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
36282025-03-08T00:49:30.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36292025-03-08T00:49:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36302025-03-08T00:49:30.439ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36312025-03-08T00:49:30.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36322025-03-08T00:49:30.443ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36332025-03-08T00:49:30.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36342025-03-08T00:49:30.447ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36352025-03-08T00:49:30.449ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
36362025-03-08T00:49:30.449ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
36372025-03-08T00:49:30.449ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
36382025-03-08T00:49:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
36392025-03-08T00:49:30.450ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 157.864695ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
36402025-03-08T00:49:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
36412025-03-08T00:49:30.451ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36422025-03-08T00:49:30.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36432025-03-08T00:49:30.454ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 161.358751ms iteration = 19 status = 0
36442025-03-08T00:49:30.463ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36452025-03-08T00:49:30.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36462025-03-08T00:49:30.465ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
36472025-03-08T00:49:30.465ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36482025-03-08T00:49:30.468ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
36492025-03-08T00:49:30.468ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36502025-03-08T00:49:30.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36512025-03-08T00:49:30.469ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
36522025-03-08T00:49:30.469ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
36532025-03-08T00:49:30.469ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f857197b-d2c8-4343-b73e-30663e8d9f92", "content-length": "26", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
36542025-03-08T00:49:30.469ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
36552025-03-08T00:49:30.472ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
36562025-03-08T00:49:30.473ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
36572025-03-08T00:49:30.473ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
36582025-03-08T00:49:30.473ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36592025-03-08T00:49:30.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36602025-03-08T00:49:30.485ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36612025-03-08T00:49:30.490ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
36622025-03-08T00:49:30.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
36632025-03-08T00:49:30.490ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
36642025-03-08T00:49:30.490ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36652025-03-08T00:49:30.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36662025-03-08T00:49:30.495ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
36672025-03-08T00:49:30.495ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
36682025-03-08T00:49:30.495ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-08 00:40:30.694594 UTC dns_group = internal generation = 1
36692025-03-08T00:49:30.495ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 203.687606ms iteration = 10 status = {"generation":1}
36702025-03-08T00:49:30.500ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
36712025-03-08T00:49:30.500ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 0 version = 2
36722025-03-08T00:49:30.500ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36732025-03-08T00:49:30.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36742025-03-08T00:49:30.502ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
36752025-03-08T00:49:30.502ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
36762025-03-08T00:49:30.502ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 209.824717ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
36772025-03-08T00:49:30.502ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36782025-03-08T00:49:30.503ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
36792025-03-08T00:49:30.503ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 210.818472ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
36802025-03-08T00:49:30.506ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36812025-03-08T00:49:30.506ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36822025-03-08T00:49:30.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36832025-03-08T00:49:30.509ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 1 version = 2
36842025-03-08T00:49:30.509ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
36852025-03-08T00:49:30.509ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-08 00:40:32.180226 UTC dns_group = external generation = 2
36862025-03-08T00:49:30.509ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 217.075477ms iteration = 10 status = {"generation":2}
36872025-03-08T00:49:30.509ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 216.476511ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
36882025-03-08T00:49:30.509ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
36892025-03-08T00:49:30.509ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
36902025-03-08T00:49:30.509ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8adbc444-c6d6-489d-bee5-5c3d01455646", "content-length": "26", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
36912025-03-08T00:49:30.509ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
36922025-03-08T00:49:30.511ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
36932025-03-08T00:49:30.527ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-08T00:40:26.976319Z, time_modified: 2025-03-08T00:40:26.976319Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }) }
36942025-03-08T00:49:30.527ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 235.038229ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]}},"default_endpoint":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
36952025-03-08T00:49:30.555ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 262.321283ms iteration = 20 status = {"physical_disks_added":0}
36962025-03-08T00:49:30.608ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
36972025-03-08T00:49:30.608ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
36982025-03-08T00:49:30.610ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36992025-03-08T00:49:30.612ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
37002025-03-08T00:49:30.612ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37012025-03-08T00:49:30.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37022025-03-08T00:49:30.613ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
37032025-03-08T00:49:30.621ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
37042025-03-08T00:49:30.621ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37052025-03-08T00:49:30.622ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "03c09a33-4d72-4de8-8a35-74e2428ccff4", "content-length": "26", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37062025-03-08T00:49:30.622ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37072025-03-08T00:49:30.623ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
37082025-03-08T00:49:30.628ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
37092025-03-08T00:49:30.628ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "67ae4010-c83c-4922-afff-9e0030c8996d", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37102025-03-08T00:49:30.628ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37112025-03-08T00:49:30.630ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "8d2e5276-9e5a-4996-aeb1-a416b896a5d2", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37122025-03-08T00:49:30.630ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37132025-03-08T00:49:30.632ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37142025-03-08T00:49:30.632ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 340.572663ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
37152025-03-08T00:49:30.638ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 346.49629ms iteration = 19 status = {}
37162025-03-08T00:49:30.659ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37172025-03-08T00:49:30.665ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
37182025-03-08T00:49:30.665ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37192025-03-08T00:49:30.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37202025-03-08T00:49:30.667ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
37212025-03-08T00:49:30.667ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
37222025-03-08T00:49:30.668ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ac3a70-9122-48d5-81ef-dc6353c7d652", "content-length": "372", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37232025-03-08T00:49:30.668ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37242025-03-08T00:49:30.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37252025-03-08T00:49:30.681ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
37262025-03-08T00:49:30.682ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c9b218e-e9b7-40c4-aa64-a4a66c12da2f", "content-length": "1", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37272025-03-08T00:49:30.684ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
37282025-03-08T00:49:30.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
37292025-03-08T00:49:30.690ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 398.257867ms iteration = 19 status = null
37302025-03-08T00:49:30.715ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
37312025-03-08T00:49:30.715ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "c5ede013-2e24-4904-9e5a-764c4be8eeb9", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
37322025-03-08T00:49:30.715ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
37332025-03-08T00:49:30.718ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "af08cca4-a3c6-43cd-b86f-9e6aac49313b", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
37342025-03-08T00:49:30.718ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
37352025-03-08T00:49:30.725ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "7aab6ba4-23b4-43e8-8a7d-8bfa4ff1a599", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
37362025-03-08T00:49:30.725ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
37372025-03-08T00:49:30.728ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
37382025-03-08T00:49:30.728ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 436.597392ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
37392025-03-08T00:49:30.761ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
37402025-03-08T00:49:30.761ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
37412025-03-08T00:49:30.761ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8075de41-fc62-4174-b241-b7611ebfa8b8", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37422025-03-08T00:49:30.761ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37432025-03-08T00:49:30.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37442025-03-08T00:49:30.765ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37452025-03-08T00:49:30.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37462025-03-08T00:49:30.768ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37472025-03-08T00:49:30.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37482025-03-08T00:49:30.773ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
37492025-03-08T00:49:30.773ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
37502025-03-08T00:49:30.773ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ae62776f-2aa3-4f98-8748-693bfa88f18d", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37512025-03-08T00:49:30.776ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 484.029114ms iteration = 19 status = {}
37522025-03-08T00:49:30.827ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
37532025-03-08T00:49:30.828ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa2e6901-cf88-4018-9e03-c63dc2d4d06f", "content-length": "691", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37542025-03-08T00:49:30.828ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
37552025-03-08T00:49:30.828ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
37562025-03-08T00:49:30.828ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
37572025-03-08T00:49:30.829ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 536.448637ms iteration = 19 status = {}
37582025-03-08T00:49:30.830ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37592025-03-08T00:49:30.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37602025-03-08T00:49:30.833ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
37612025-03-08T00:49:30.833ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37622025-03-08T00:49:30.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37632025-03-08T00:49:30.844ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
37642025-03-08T00:49:30.844ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "af26632a-5fd4-4f76-9ef8-2409041a0d16", "content-length": "134", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37652025-03-08T00:49:30.844ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
37662025-03-08T00:49:30.844ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
37672025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
37682025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
37692025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
37702025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
37712025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
37722025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
37732025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
37742025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
37752025-03-08T00:49:30.845ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
37762025-03-08T00:49:30.845ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37772025-03-08T00:49:30.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37782025-03-08T00:49:30.848ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
37792025-03-08T00:49:30.848ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "fda49a75-4f2a-4f7b-a68c-7d2b86c6b916", "content-length": "2", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37802025-03-08T00:49:30.848ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
37812025-03-08T00:49:30.848ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c73ca91-ba45-4fae-8334-5b6f38ac5bda", "content-length": "52", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37822025-03-08T00:49:30.848ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
37832025-03-08T00:49:30.848ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
37842025-03-08T00:49:30.974ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "7f1aad76-47a6-4044-a96b-c1666781afc7", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37852025-03-08T00:49:30.974ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
37862025-03-08T00:49:30.974ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
37872025-03-08T00:49:30.974ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "13e06188-e99e-45e6-a9fb-41c3499a1070", "content-length": "682", "date": "Sat, 08 Mar 2025 00:49:30 GMT"} })
37882025-03-08T00:49:30.974ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37892025-03-08T00:49:30.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37902025-03-08T00:49:30.978ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37912025-03-08T00:49:30.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37922025-03-08T00:49:30.981ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37932025-03-08T00:49:30.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37942025-03-08T00:49:30.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37952025-03-08T00:49:30.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37962025-03-08T00:49:30.994ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 701.769926ms iteration = 19 status = {}
37972025-03-08T00:49:32.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 384 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 91034ba0-801f-4a2e-9c05-906b0062c5a8 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
37982025-03-08T00:49:40.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
37992025-03-08T00:49:40.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38002025-03-08T00:49:40.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38012025-03-08T00:49:40.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
38022025-03-08T00:49:40.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38032025-03-08T00:49:40.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38042025-03-08T00:49:40.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38052025-03-08T00:49:40.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38062025-03-08T00:49:40.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38072025-03-08T00:49:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
38082025-03-08T00:49:40.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38092025-03-08T00:49:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38102025-03-08T00:49:40.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38112025-03-08T00:49:40.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38122025-03-08T00:49:40.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38132025-03-08T00:49:40.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
38142025-03-08T00:49:40.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38152025-03-08T00:49:40.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38162025-03-08T00:49:40.315ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
38172025-03-08T00:49:40.315ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.028708ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
38182025-03-08T00:49:40.331ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.298465ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
38192025-03-08T00:49:42.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 311 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 9fc467c3-c374-42f6-ba8e-790225121dd2 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
38202025-03-08T00:49:50.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
38212025-03-08T00:49:50.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38222025-03-08T00:49:50.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38232025-03-08T00:49:50.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
38242025-03-08T00:49:50.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38252025-03-08T00:49:50.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38262025-03-08T00:49:50.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38272025-03-08T00:49:50.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38282025-03-08T00:49:50.296ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.913906ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
38292025-03-08T00:49:50.325ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38302025-03-08T00:49:50.328ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
38312025-03-08T00:49:50.328ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38322025-03-08T00:49:50.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38332025-03-08T00:49:50.330ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38342025-03-08T00:49:50.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38352025-03-08T00:49:50.332ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38362025-03-08T00:49:50.334ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
38372025-03-08T00:49:50.334ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38382025-03-08T00:49:50.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38392025-03-08T00:49:50.345ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
38402025-03-08T00:49:50.345ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 53.599562ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
38412025-03-08T00:49:52.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 379 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = af580eab-4e4c-4228-a222-f4c577680c7a response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
38422025-03-08T00:50:00.291ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
38432025-03-08T00:50:00.291ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
38442025-03-08T00:50:00.291ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
38452025-03-08T00:50:00.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
38462025-03-08T00:50:00.291ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
38472025-03-08T00:50:00.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38482025-03-08T00:50:00.291ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
38492025-03-08T00:50:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
38502025-03-08T00:50:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38512025-03-08T00:50:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
38522025-03-08T00:50:00.292ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
38532025-03-08T00:50:00.292ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
38542025-03-08T00:50:00.292ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
38552025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38562025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
38572025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38582025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
38592025-03-08T00:50:00.293ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
38602025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
38612025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38622025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38632025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
38642025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
38652025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38662025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38672025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
38682025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
38692025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
38702025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
38712025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38722025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
38732025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
38742025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
38752025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
38762025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
38772025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
38782025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
38792025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
38802025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
38812025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
38822025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
38832025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
38842025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38852025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38862025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38872025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
38882025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
38892025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38902025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
38912025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
38922025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38932025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
38942025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38952025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
38962025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
38972025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38982025-03-08T00:50:00.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
38992025-03-08T00:50:00.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39002025-03-08T00:50:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1483, tv_nsec: 318802253 } })
39012025-03-08T00:50:00.295ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
39022025-03-08T00:50:00.295ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39032025-03-08T00:50:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1483, tv_nsec: 319461889 } })
39042025-03-08T00:50:00.295ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.977958ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
39052025-03-08T00:50:00.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
39062025-03-08T00:50:00.296ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39072025-03-08T00:50:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
39082025-03-08T00:50:00.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39092025-03-08T00:50:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39102025-03-08T00:50:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
39112025-03-08T00:50:00.299ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39122025-03-08T00:50:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
39132025-03-08T00:50:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
39142025-03-08T00:50:00.299ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39152025-03-08T00:50:00.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39162025-03-08T00:50:00.302ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39172025-03-08T00:50:00.302ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39182025-03-08T00:50:00.302ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39192025-03-08T00:50:00.304ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 11.017962ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
39202025-03-08T00:50:00.304ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 11.814371ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
39212025-03-08T00:50:00.305ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
39222025-03-08T00:50:00.305ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39232025-03-08T00:50:00.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39242025-03-08T00:50:00.306ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
39252025-03-08T00:50:00.306ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39262025-03-08T00:50:00.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
39272025-03-08T00:50:00.306ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
39282025-03-08T00:50:00.306ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
39292025-03-08T00:50:00.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
39302025-03-08T00:50:00.309ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39312025-03-08T00:50:00.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39322025-03-08T00:50:00.310ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39332025-03-08T00:50:00.310ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39342025-03-08T00:50:00.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39352025-03-08T00:50:00.311ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39362025-03-08T00:50:00.312ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
39372025-03-08T00:50:00.312ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39382025-03-08T00:50:00.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39392025-03-08T00:50:00.315ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
39402025-03-08T00:50:00.315ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39412025-03-08T00:50:00.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39422025-03-08T00:50:00.318ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39432025-03-08T00:50:00.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
39442025-03-08T00:50:00.319ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
39452025-03-08T00:50:00.319ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 26.290909ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
39462025-03-08T00:50:00.319ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39472025-03-08T00:50:00.322ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
39482025-03-08T00:50:00.322ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
39492025-03-08T00:50:00.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3950 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
39512025-03-08T00:50:00.347ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39522025-03-08T00:50:00.347ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.868114ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
39532025-03-08T00:50:00.349ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
39542025-03-08T00:50:00.349ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.348135ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
39552025-03-08T00:50:00.351ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39562025-03-08T00:50:00.351ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
39572025-03-08T00:50:00.351ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39582025-03-08T00:50:00.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39592025-03-08T00:50:00.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39602025-03-08T00:50:00.352ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39612025-03-08T00:50:00.352ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
39622025-03-08T00:50:00.352ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
39632025-03-08T00:50:00.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
39642025-03-08T00:50:00.354ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39652025-03-08T00:50:00.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39662025-03-08T00:50:00.355ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1483, tv_nsec: 379609893 } })
39672025-03-08T00:50:00.356ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
39682025-03-08T00:50:00.357ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39692025-03-08T00:50:00.357ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39702025-03-08T00:50:00.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39712025-03-08T00:50:00.361ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
39722025-03-08T00:50:00.361ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
39732025-03-08T00:50:00.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
39742025-03-08T00:50:00.362ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39752025-03-08T00:50:00.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39762025-03-08T00:50:00.364ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
39772025-03-08T00:50:00.364ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 71.198247ms iteration = 20 status = 0
39782025-03-08T00:50:00.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
39792025-03-08T00:50:00.364ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 71.782252ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39802025-03-08T00:50:00.365ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39812025-03-08T00:50:00.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
39822025-03-08T00:50:00.366ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 73.028502ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
39832025-03-08T00:50:00.366ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39842025-03-08T00:50:00.367ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
39852025-03-08T00:50:00.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
39862025-03-08T00:50:00.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
39872025-03-08T00:50:00.369ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
39882025-03-08T00:50:00.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
39892025-03-08T00:50:00.371ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
39902025-03-08T00:50:00.371ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 78.332843ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
39912025-03-08T00:50:00.407ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
39922025-03-08T00:50:00.407ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
39932025-03-08T00:50:00.407ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
39942025-03-08T00:50:00.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
39952025-03-08T00:50:00.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
39962025-03-08T00:50:00.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
39972025-03-08T00:50:00.423ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 130.695194ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
39982025-03-08T00:50:00.429ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39992025-03-08T00:50:00.429ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
40002025-03-08T00:50:00.429ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
40012025-03-08T00:50:00.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40022025-03-08T00:50:00.429ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecbfacfb-614c-4b21-847f-980133cb9730", "content-length": "26", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40032025-03-08T00:50:00.429ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
40042025-03-08T00:50:00.432ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
40052025-03-08T00:50:00.447ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
40062025-03-08T00:50:00.447ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
40072025-03-08T00:50:00.448ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3a4b43a-69d4-4854-aa55-45f9d4439e62", "content-length": "26", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40082025-03-08T00:50:00.448ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
40092025-03-08T00:50:00.450ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
40102025-03-08T00:50:00.461ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
40112025-03-08T00:50:00.461ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
40122025-03-08T00:50:00.461ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 168.340614ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
40132025-03-08T00:50:00.464ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 171.316718ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
40142025-03-08T00:50:00.468ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
40152025-03-08T00:50:00.468ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
40162025-03-08T00:50:00.468ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "66497fef-3db8-484d-8924-7af6978b87b7", "content-length": "26", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40172025-03-08T00:50:00.468ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
40182025-03-08T00:50:00.470ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
40192025-03-08T00:50:00.480ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 187.962938ms iteration = 21 status = {"physical_disks_added":0}
40202025-03-08T00:50:00.513ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
40212025-03-08T00:50:00.513ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
40222025-03-08T00:50:00.514ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1171c2e-d782-41f8-b238-f26e5725f83b", "content-length": "372", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40232025-03-08T00:50:00.514ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40242025-03-08T00:50:00.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40252025-03-08T00:50:00.545ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
40262025-03-08T00:50:00.546ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2ae6bb4-59c5-488c-8e5b-aa92224d230a", "content-length": "1", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40272025-03-08T00:50:00.548ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40282025-03-08T00:50:00.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40292025-03-08T00:50:00.550ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 258.98578ms iteration = 20 status = null
40302025-03-08T00:50:00.561ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 269.669974ms iteration = 20 status = {}
40312025-03-08T00:50:00.568ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40322025-03-08T00:50:00.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40332025-03-08T00:50:00.570ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40342025-03-08T00:50:00.571ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
40352025-03-08T00:50:00.571ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
40362025-03-08T00:50:00.572ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "940475f0-0794-47d7-af8c-3682049750e6", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40372025-03-08T00:50:00.572ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40382025-03-08T00:50:00.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40392025-03-08T00:50:00.573ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
40402025-03-08T00:50:00.573ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40412025-03-08T00:50:00.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40422025-03-08T00:50:00.575ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40432025-03-08T00:50:00.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40442025-03-08T00:50:00.579ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40452025-03-08T00:50:00.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40462025-03-08T00:50:00.582ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
40472025-03-08T00:50:00.582ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
40482025-03-08T00:50:00.583ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "daa59874-d426-415b-89fb-c13f17e429b0", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40492025-03-08T00:50:00.585ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 292.588809ms iteration = 20 status = {}
40502025-03-08T00:50:00.612ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40512025-03-08T00:50:00.615ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
40522025-03-08T00:50:00.615ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40532025-03-08T00:50:00.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40542025-03-08T00:50:00.655ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40552025-03-08T00:50:00.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40562025-03-08T00:50:00.658ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
40572025-03-08T00:50:00.658ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40582025-03-08T00:50:00.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40592025-03-08T00:50:00.675ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
40602025-03-08T00:50:00.675ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c17cd872-a0b9-49e5-905e-92045d9e4aa5", "content-length": "134", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40612025-03-08T00:50:00.675ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
40622025-03-08T00:50:00.675ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
40632025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
40642025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
40652025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40662025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40672025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40682025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
40692025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
40702025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
40712025-03-08T00:50:00.676ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
40722025-03-08T00:50:00.676ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40732025-03-08T00:50:00.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40742025-03-08T00:50:00.679ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
40752025-03-08T00:50:00.679ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8af6ae7-144c-4aab-9a6f-cee93bdf28e1", "content-length": "2", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40762025-03-08T00:50:00.679ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
40772025-03-08T00:50:00.679ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a09440e-2b2e-43b8-bb37-4b4cf03006cd", "content-length": "52", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40782025-03-08T00:50:00.679ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
40792025-03-08T00:50:00.679ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
40802025-03-08T00:50:00.737ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
40812025-03-08T00:50:00.791ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "200568e1-dec7-400b-bbb5-b3479c12345f", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40822025-03-08T00:50:00.791ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
40832025-03-08T00:50:00.791ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
40842025-03-08T00:50:00.792ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "185fa531-711f-4785-8e8c-044978573425", "content-length": "682", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40852025-03-08T00:50:00.792ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40862025-03-08T00:50:00.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40872025-03-08T00:50:00.792ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "56ce463b-d50f-495b-a89a-7eacf411950b", "content-length": "691", "date": "Sat, 08 Mar 2025 00:50:00 GMT"} })
40882025-03-08T00:50:00.792ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
40892025-03-08T00:50:00.792ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
40902025-03-08T00:50:00.792ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
40912025-03-08T00:50:00.794ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 501.247398ms iteration = 20 status = {}
40922025-03-08T00:50:00.796ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40932025-03-08T00:50:00.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40942025-03-08T00:50:00.799ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40952025-03-08T00:50:00.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40962025-03-08T00:50:00.802ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40972025-03-08T00:50:00.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40982025-03-08T00:50:00.809ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 516.865384ms iteration = 20 status = {}
40992025-03-08T00:50:02.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 339 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 4d63669c-b42f-4849-8573-bfec6b558b75 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
41002025-03-08T00:50:10.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
41012025-03-08T00:50:10.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41022025-03-08T00:50:10.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41032025-03-08T00:50:10.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
41042025-03-08T00:50:10.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41052025-03-08T00:50:10.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41062025-03-08T00:50:10.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41072025-03-08T00:50:10.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41082025-03-08T00:50:10.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41092025-03-08T00:50:10.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41102025-03-08T00:50:10.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41112025-03-08T00:50:10.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41122025-03-08T00:50:10.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41132025-03-08T00:50:10.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41142025-03-08T00:50:10.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41152025-03-08T00:50:10.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
41162025-03-08T00:50:10.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41172025-03-08T00:50:10.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41182025-03-08T00:50:10.312ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
41192025-03-08T00:50:10.312ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.243794ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
41202025-03-08T00:50:10.321ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 29.235591ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41212025-03-08T00:50:12.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 355 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = d2bdad21-2fa7-45c7-9b6b-bfb97ac518ef response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
41222025-03-08T00:50:20.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
41232025-03-08T00:50:20.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41242025-03-08T00:50:20.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41252025-03-08T00:50:20.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
41262025-03-08T00:50:20.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41272025-03-08T00:50:20.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41282025-03-08T00:50:20.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41292025-03-08T00:50:20.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41302025-03-08T00:50:20.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41312025-03-08T00:50:20.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41322025-03-08T00:50:20.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41332025-03-08T00:50:20.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41342025-03-08T00:50:20.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41352025-03-08T00:50:20.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41362025-03-08T00:50:20.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41372025-03-08T00:50:20.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
41382025-03-08T00:50:20.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41392025-03-08T00:50:20.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41402025-03-08T00:50:20.312ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
41412025-03-08T00:50:20.312ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.809184ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
41422025-03-08T00:50:20.323ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.349293ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41432025-03-08T00:50:22.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 361 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = e876c5f3-c66e-4333-996e-1a4414a4eae5 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
41442025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
41452025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
41462025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
41472025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
41482025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
41492025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
41502025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
41512025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
41522025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
41532025-03-08T00:50:30.292ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = a346f089-8033-4b92-b854-361d1da5cab2
41542025-03-08T00:50:30.292ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-08 00:40:30.292127107 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
41552025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 19.341\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"a346f089-8033-4b92-b854-361d1da5cab2"}
41562025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
41572025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
41582025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
41592025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
41602025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
41612025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
41622025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
41632025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41642025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
41652025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
41662025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
41672025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
41682025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
41692025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
41702025-03-08T00:50:30.292ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
41712025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
41722025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
41732025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41742025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
41752025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
41762025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
41772025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
41782025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
41792025-03-08T00:50:30.292ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
41802025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
41812025-03-08T00:50:30.292ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
41822025-03-08T00:50:30.292ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
41832025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
41842025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41852025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
41862025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41872025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
41882025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
41892025-03-08T00:50:30.292ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
41902025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
41912025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
41922025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41932025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
41942025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
41952025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
41962025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
41972025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
41982025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41992025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
42002025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
42012025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
42022025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
42032025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42042025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
42052025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
42062025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
42072025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
42082025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
42092025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42102025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
42112025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42122025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
42132025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
42142025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
42152025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
42162025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
42172025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
42182025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
42192025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
42202025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42212025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42222025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
42232025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
42242025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
42252025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42262025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42272025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
42282025-03-08T00:50:30.292ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("59287b7e-0d61-4586-b7c5-e79476927737.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1513, tv_nsec: 316788053 } })
42292025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
42302025-03-08T00:50:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42312025-03-08T00:50:30.293ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
42322025-03-08T00:50:30.293ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1ac98a83-1243-46f5-99db-fd29222026d8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("508de01a-93ec-4309-a59e-0cfd22b5027a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1513, tv_nsec: 316882465 } })
42332025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
42342025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
42352025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42362025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
42372025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42382025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
42392025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
42402025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
42412025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42422025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
42432025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
42442025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
42452025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
42462025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
42472025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
42482025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42492025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
42502025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
42512025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42522025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42532025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
42542025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
42552025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.332533ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
42562025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42572025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
42582025-03-08T00:50:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.382824ms iteration = 11 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
42592025-03-08T00:50:30.294ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
42602025-03-08T00:50:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
42612025-03-08T00:50:30.295ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-08 00:40:30.292127107 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
42622025-03-08T00:50:30.295ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.894051ms iteration = 11 status = {"expiration":"2025-03-08T00:40:30.292127107Z","pruned":[]}
42632025-03-08T00:50:30.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1513, tv_nsec: 319709875 } })
42642025-03-08T00:50:30.295ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42652025-03-08T00:50:30.296ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
42662025-03-08T00:50:30.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
42672025-03-08T00:50:30.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
42682025-03-08T00:50:30.296ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.204114ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
42692025-03-08T00:50:30.297ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42702025-03-08T00:50:30.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
42712025-03-08T00:50:30.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42722025-03-08T00:50:30.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42732025-03-08T00:50:30.300ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
42742025-03-08T00:50:30.300ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42752025-03-08T00:50:30.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42762025-03-08T00:50:30.302ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 9.579526ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
42772025-03-08T00:50:30.302ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42782025-03-08T00:50:30.302ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42792025-03-08T00:50:30.303ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
42802025-03-08T00:50:30.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
42812025-03-08T00:50:30.303ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1513, tv_nsec: 327780784 } })
42822025-03-08T00:50:30.304ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
42832025-03-08T00:50:30.305ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
42842025-03-08T00:50:30.305ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
42852025-03-08T00:50:30.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
42862025-03-08T00:50:30.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
42872025-03-08T00:50:30.307ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42882025-03-08T00:50:30.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42892025-03-08T00:50:30.308ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42902025-03-08T00:50:30.311ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
42912025-03-08T00:50:30.311ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
42922025-03-08T00:50:30.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
42932025-03-08T00:50:30.314ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
42942025-03-08T00:50:30.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
42952025-03-08T00:50:30.315ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42962025-03-08T00:50:30.318ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
42972025-03-08T00:50:30.318ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
42982025-03-08T00:50:30.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
42992025-03-08T00:50:30.324ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43002025-03-08T00:50:30.327ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
43012025-03-08T00:50:30.337ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43022025-03-08T00:50:30.341ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
43032025-03-08T00:50:30.353ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43042025-03-08T00:50:30.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43052025-03-08T00:50:30.357ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43062025-03-08T00:50:30.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43072025-03-08T00:50:30.361ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43082025-03-08T00:50:30.362ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43092025-03-08T00:50:30.363ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43102025-03-08T00:50:30.366ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
43112025-03-08T00:50:30.366ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43122025-03-08T00:50:30.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43132025-03-08T00:50:30.367ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
43142025-03-08T00:50:30.367ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
43152025-03-08T00:50:30.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
43162025-03-08T00:50:30.368ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
43172025-03-08T00:50:30.368ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
43182025-03-08T00:50:30.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
43192025-03-08T00:50:30.370ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43202025-03-08T00:50:30.370ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
43212025-03-08T00:50:30.370ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43222025-03-08T00:50:30.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
43232025-03-08T00:50:30.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43242025-03-08T00:50:30.370ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.359853ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
43252025-03-08T00:50:30.371ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 79.398419ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
43262025-03-08T00:50:30.373ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
43272025-03-08T00:50:30.373ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
43282025-03-08T00:50:30.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
43292025-03-08T00:50:30.374ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
43302025-03-08T00:50:30.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
43312025-03-08T00:50:30.374ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
43322025-03-08T00:50:30.374ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 82.581288ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
43332025-03-08T00:50:30.376ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
43342025-03-08T00:50:30.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
43352025-03-08T00:50:30.377ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43362025-03-08T00:50:30.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43372025-03-08T00:50:30.379ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43382025-03-08T00:50:30.380ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
43392025-03-08T00:50:30.380ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 87.712945ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
43402025-03-08T00:50:30.382ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
43412025-03-08T00:50:30.382ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
43422025-03-08T00:50:30.382ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 90.35783ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
43432025-03-08T00:50:30.383ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
43442025-03-08T00:50:30.383ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
43452025-03-08T00:50:30.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
43462025-03-08T00:50:30.383ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
43472025-03-08T00:50:30.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
43482025-03-08T00:50:30.384ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
43492025-03-08T00:50:30.384ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 92.236536ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
43502025-03-08T00:50:30.385ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4351 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
43522025-03-08T00:50:30.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
43532025-03-08T00:50:30.386ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43542025-03-08T00:50:30.387ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
43552025-03-08T00:50:30.387ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
43562025-03-08T00:50:30.387ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
43572025-03-08T00:50:30.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
43582025-03-08T00:50:30.389ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
43592025-03-08T00:50:30.389ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43602025-03-08T00:50:30.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43612025-03-08T00:50:30.392ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 99.807772ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
43622025-03-08T00:50:30.393ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
43632025-03-08T00:50:30.393ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
43642025-03-08T00:50:30.393ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
43652025-03-08T00:50:30.393ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43662025-03-08T00:50:30.395ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43672025-03-08T00:50:30.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43682025-03-08T00:50:30.396ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43692025-03-08T00:50:30.397ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
43702025-03-08T00:50:30.397ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43712025-03-08T00:50:30.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43722025-03-08T00:50:30.399ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
43732025-03-08T00:50:30.399ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
43742025-03-08T00:50:30.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
43752025-03-08T00:50:30.405ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
43762025-03-08T00:50:30.405ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
43772025-03-08T00:50:30.405ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
43782025-03-08T00:50:30.405ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43792025-03-08T00:50:30.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43802025-03-08T00:50:30.410ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
43812025-03-08T00:50:30.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
43822025-03-08T00:50:30.424ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43832025-03-08T00:50:30.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
43842025-03-08T00:50:30.428ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43852025-03-08T00:50:30.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43862025-03-08T00:50:30.433ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
43872025-03-08T00:50:30.443ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
43882025-03-08T00:50:30.443ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
43892025-03-08T00:50:30.443ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 151.276163ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
43902025-03-08T00:50:30.448ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 156.05696ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43912025-03-08T00:50:30.471ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
43922025-03-08T00:50:30.503ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
43932025-03-08T00:50:30.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
43942025-03-08T00:50:30.509ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
43952025-03-08T00:50:30.509ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
43962025-03-08T00:50:30.510ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c83aa51-2120-4df3-ab49-c8ccddf422e0", "content-length": "372", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
43972025-03-08T00:50:30.510ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
43982025-03-08T00:50:30.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
43992025-03-08T00:50:30.512ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
44002025-03-08T00:50:30.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
44012025-03-08T00:50:30.513ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
44022025-03-08T00:50:30.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
44032025-03-08T00:50:30.514ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44042025-03-08T00:50:30.516ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
44052025-03-08T00:50:30.516ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
44062025-03-08T00:50:30.516ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3cdc68-0299-4649-9835-b8807a225b2f", "content-length": "26", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
44072025-03-08T00:50:30.516ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
44082025-03-08T00:50:30.517ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
44092025-03-08T00:50:30.517ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44102025-03-08T00:50:30.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44112025-03-08T00:50:30.518ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
44122025-03-08T00:50:30.520ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44132025-03-08T00:50:30.522ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44142025-03-08T00:50:30.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44152025-03-08T00:50:30.522ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1513, tv_nsec: 546270585 } })
44162025-03-08T00:50:30.522ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
44172025-03-08T00:50:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
44182025-03-08T00:50:30.524ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44192025-03-08T00:50:30.524ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44202025-03-08T00:50:30.527ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44212025-03-08T00:50:30.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44222025-03-08T00:50:30.533ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(8505427c-92d8-4d72-ac7c-a1f22f744274, 0), (187dedeb-5992-43cb-87d1-04ff54604b8f, 0), (90f8ace6-36ae-4a38-9dee-2c75ab9a1802, 0), (5d660b6a-29ce-4cae-9812-4763cac396ab, 0)] collection_id = 8505427c-92d8-4d72-ac7c-a1f22f744274
44232025-03-08T00:50:30.533ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44242025-03-08T00:50:30.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44252025-03-08T00:50:30.539ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
44262025-03-08T00:50:30.539ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
44272025-03-08T00:50:30.539ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "75386c68-93e9-432b-9f72-9ee50d186703", "content-length": "26", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
44282025-03-08T00:50:30.539ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
4429 Batch: [InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 5d660b6a-29ce-4cae-9812-4763cac396ab (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
44302025-03-08T00:50:30.544ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
44312025-03-08T00:50:30.552ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44322025-03-08T00:50:30.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44332025-03-08T00:50:30.554ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44342025-03-08T00:50:30.557ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
44352025-03-08T00:50:30.557ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44362025-03-08T00:50:30.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44372025-03-08T00:50:30.558ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
44382025-03-08T00:50:30.558ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 266.143867ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
44392025-03-08T00:50:30.559ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
44402025-03-08T00:50:30.559ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 266.908105ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
44412025-03-08T00:50:30.560ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 267.51235ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
44422025-03-08T00:50:30.568ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
44432025-03-08T00:50:30.568ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
44442025-03-08T00:50:30.568ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44452025-03-08T00:50:30.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
44462025-03-08T00:50:30.569ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
44472025-03-08T00:50:30.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
44482025-03-08T00:50:30.570ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44492025-03-08T00:50:30.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
44502025-03-08T00:50:30.574ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44512025-03-08T00:50:30.575ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
44522025-03-08T00:50:30.575ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 283.577247ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
44532025-03-08T00:50:30.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
44542025-03-08T00:50:30.576ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44552025-03-08T00:50:30.577ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44562025-03-08T00:50:30.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44572025-03-08T00:50:30.578ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
44582025-03-08T00:50:30.578ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
44592025-03-08T00:50:30.578ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44602025-03-08T00:50:30.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
44612025-03-08T00:50:30.579ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44622025-03-08T00:50:30.580ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44632025-03-08T00:50:30.580ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44642025-03-08T00:50:30.581ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
44652025-03-08T00:50:30.581ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44662025-03-08T00:50:30.581ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44672025-03-08T00:50:30.583ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
44682025-03-08T00:50:30.583ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44692025-03-08T00:50:30.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44702025-03-08T00:50:30.583ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
44712025-03-08T00:50:30.583ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44722025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
44732025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
44742025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44752025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44762025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
44772025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44782025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44792025-03-08T00:50:30.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44802025-03-08T00:50:30.585ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
44812025-03-08T00:50:30.585ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
44822025-03-08T00:50:30.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
44832025-03-08T00:50:30.587ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44842025-03-08T00:50:30.588ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
44852025-03-08T00:50:30.588ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 0 version = 2
44862025-03-08T00:50:30.588ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44872025-03-08T00:50:30.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44882025-03-08T00:50:30.589ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 297.05778ms iteration = 21 status = 0
44892025-03-08T00:50:30.589ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44902025-03-08T00:50:30.592ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
44912025-03-08T00:50:30.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44922025-03-08T00:50:30.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44932025-03-08T00:50:30.593ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
44942025-03-08T00:50:30.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
44952025-03-08T00:50:30.593ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
44962025-03-08T00:50:30.593ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44972025-03-08T00:50:30.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44982025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44992025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
45002025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
45012025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-08 00:40:30.694594 UTC dns_group = internal generation = 1
45022025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 1 version = 2
45032025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
45042025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-08 00:40:32.180226 UTC dns_group = external generation = 2
45052025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 303.353985ms iteration = 11 status = {"generation":1}
45062025-03-08T00:50:30.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 303.475348ms iteration = 11 status = {"generation":2}
45072025-03-08T00:50:30.597ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
45082025-03-08T00:50:30.597ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
45092025-03-08T00:50:30.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
45102025-03-08T00:50:30.598ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
45112025-03-08T00:50:30.598ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
45122025-03-08T00:50:30.598ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "25656ef9-75e0-4448-ab1a-f5409530c07d", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
45132025-03-08T00:50:30.598ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45142025-03-08T00:50:30.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45152025-03-08T00:50:30.598ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45162025-03-08T00:50:30.602ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45172025-03-08T00:50:30.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45182025-03-08T00:50:30.604ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
45192025-03-08T00:50:30.604ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "48a8d78b-639e-4b4b-b88b-09ad31ca68f5", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
45202025-03-08T00:50:30.604ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
45212025-03-08T00:50:30.604ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
45222025-03-08T00:50:30.604ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
45232025-03-08T00:50:30.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
45242025-03-08T00:50:30.606ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "be00436a-56d4-461e-ac24-da3aacec0c00", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
45252025-03-08T00:50:30.606ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
45262025-03-08T00:50:30.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45272025-03-08T00:50:30.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45282025-03-08T00:50:30.607ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
45292025-03-08T00:50:30.607ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45302025-03-08T00:50:30.607ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
45312025-03-08T00:50:30.607ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 315.372592ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
45322025-03-08T00:50:30.607ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 315.486475ms iteration = 4 status = {"blueprint_id":"a346f089-8033-4b92-b854-361d1da5cab2","inventory_collection_id":"5d660b6a-29ce-4cae-9812-4763cac396ab","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
45332025-03-08T00:50:30.608ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
45342025-03-08T00:50:30.608ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 316.789947ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
45352025-03-08T00:50:30.612ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
45362025-03-08T00:50:30.618ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45372025-03-08T00:50:30.620ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 328.309401ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
45382025-03-08T00:50:30.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
45392025-03-08T00:50:30.621ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
45402025-03-08T00:50:30.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
45412025-03-08T00:50:30.631ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-08T00:40:26.976319Z, time_modified: 2025-03-08T00:40:26.976319Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }) }
45422025-03-08T00:50:30.631ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 339.377894ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]}},"default_endpoint":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
45432025-03-08T00:50:30.651ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = 8505427c-92d8-4d72-ac7c-a1f22f744274 file = nexus/db-queries/src/db/datastore/inventory.rs:1447 ncabooses = 12 nclickhouse_keeper_membership = 0 ncollections = 1 ndatasets = 84 nerrors = 0 nnics = 6 nnvme_disk_firmware = 11 nphysical_disks = 11 nrot_pages = 8 nrots = 2 nsled_agent_zones = 1 nsled_agents = 1 nsps = 2 nzones = 28 nzpools = 9
45442025-03-08T00:50:30.651ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45452025-03-08T00:50:30.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45462025-03-08T00:50:30.673ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45472025-03-08T00:50:30.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45482025-03-08T00:50:30.677ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
45492025-03-08T00:50:30.677ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0f05b33-81c1-42cf-ba56-13aace980542", "content-length": "1", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
45502025-03-08T00:50:30.680ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
45512025-03-08T00:50:30.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
45522025-03-08T00:50:30.693ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 400.938063ms iteration = 21 status = {}
45532025-03-08T00:50:30.699ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
45542025-03-08T00:50:30.699ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
45552025-03-08T00:50:30.699ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1552c80-ab76-43f1-9b65-68a34ae0e63e", "content-length": "26", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
45562025-03-08T00:50:30.699ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
45572025-03-08T00:50:30.703ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
45582025-03-08T00:50:30.718ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
45592025-03-08T00:50:30.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
45602025-03-08T00:50:30.721ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 428.630536ms iteration = 22 status = {"physical_disks_added":0}
45612025-03-08T00:50:30.723ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45622025-03-08T00:50:30.725ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 433.108537ms iteration = 21 status = null
45632025-03-08T00:50:30.727ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
45642025-03-08T00:50:30.727ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45652025-03-08T00:50:30.730ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
45662025-03-08T00:50:30.730ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
45672025-03-08T00:50:30.731ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "af08cdcf-430c-4a83-965b-3fa0408a3a56", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
45682025-03-08T00:50:30.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45692025-03-08T00:50:30.733ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 440.798566ms iteration = 21 status = {}
45702025-03-08T00:50:30.734ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(187dedeb-5992-43cb-87d1-04ff54604b8f, 0), (90f8ace6-36ae-4a38-9dee-2c75ab9a1802, 0), (5d660b6a-29ce-4cae-9812-4763cac396ab, 0)]
45712025-03-08T00:50:30.735ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1513, tv_nsec: 759412414 } })
45722025-03-08T00:50:30.743ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45732025-03-08T00:50:30.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
45742025-03-08T00:50:30.747ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45752025-03-08T00:50:30.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
45762025-03-08T00:50:30.764ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
45772025-03-08T00:50:30.765ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5a8053e6-1453-448d-b8e7-4a584130e88d", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
45782025-03-08T00:50:30.765ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
45792025-03-08T00:50:30.767ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "07d5fc15-2d65-4a2b-addd-545ae0dfc55f", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
45802025-03-08T00:50:30.767ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
45812025-03-08T00:50:30.770ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "84981a9f-f8a1-4f43-a5f2-3cdc7b8c3a05", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
45822025-03-08T00:50:30.770ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
45832025-03-08T00:50:30.773ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
45842025-03-08T00:50:30.773ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 481.393298ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
45852025-03-08T00:50:30.779ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
45862025-03-08T00:50:30.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
45872025-03-08T00:50:30.780ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45882025-03-08T00:50:30.781ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
45892025-03-08T00:50:30.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
45902025-03-08T00:50:30.784ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
45912025-03-08T00:50:30.784ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
45922025-03-08T00:50:30.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
45932025-03-08T00:50:30.810ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
45942025-03-08T00:50:30.810ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "8786a610-9f54-4438-ac14-76fc64d8cef4", "content-length": "2", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
45952025-03-08T00:50:30.810ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:615 sled = http://[fd00:1122:3344:101::1]:12345
45962025-03-08T00:50:30.814ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 521.810746ms iteration = 3 status = {"last_run_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"delete_err":0,"delete_ok":0,"list_err":0,"list_ok":3,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"operation":"list","target_sled":"87826585-0843-40d2-a3ce-382fa54dbc5c","time":"2025-03-08T00:40:30.730769703Z"},{"operation":"list","target_sled":"87826585-0843-40d2-a3ce-382fa54dbc5c","time":"2025-03-08T00:45:30.647352828Z"},{"operation":"list","target_sled":"87826585-0843-40d2-a3ce-382fa54dbc5c","time":"2025-03-08T00:50:30.810929468Z"}]}
45972025-03-08T00:50:30.842ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
45982025-03-08T00:50:30.842ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
45992025-03-08T00:50:30.842ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
46002025-03-08T00:50:30.842ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d4f960b-7b2a-4af6-8851-f38b1d8cbf3b", "content-length": "476", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46012025-03-08T00:50:30.842ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
46022025-03-08T00:50:30.842ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2115fe9d-4345-45bc-9eae-63c95a4ac225", "content-length": "734", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46032025-03-08T00:50:30.843ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
46042025-03-08T00:50:30.843ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46052025-03-08T00:50:30.843ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6da8ba0d-fe97-472b-b282-97596d3fe5c3", "content-length": "111", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46062025-03-08T00:50:30.843ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
46072025-03-08T00:50:30.843ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98982e42-e140-4be9-9e1d-5289383f8929", "content-length": "111", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46082025-03-08T00:50:30.843ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
46092025-03-08T00:50:30.844ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb3efaf7-e91e-48b6-adb3-daaf3a277aec", "content-length": "167", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46102025-03-08T00:50:30.844ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
46112025-03-08T00:50:30.844ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3f30c2-046e-4263-929e-cd139edbdd1c", "content-length": "167", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46122025-03-08T00:50:30.844ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
46132025-03-08T00:50:30.844ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "95585694-a877-4b64-bcc2-55233df6cb23", "content-length": "175", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46142025-03-08T00:50:30.845ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
46152025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df8db95d-8f87-4799-bdc4-2a68a8d713e1", "content-length": "175", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46162025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
46172025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e04dcf9c-a0bb-4c92-868b-a8e5ec5b6e19", "content-length": "702", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46182025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
46192025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b755beb-74f3-49ae-bf40-c2f5a630bcbd", "content-length": "727", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46202025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
46212025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f123ea8-d0b7-4c1f-b365-ee6cb5c9e33a", "content-length": "729", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46222025-03-08T00:50:30.846ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
46232025-03-08T00:50:30.847ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d70d5b66-b3e5-49ee-9a04-7913f692c033", "content-length": "728", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46242025-03-08T00:50:30.847ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
46252025-03-08T00:50:30.847ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a6c0632-0029-4c7a-a989-1b7743e84984", "content-length": "722", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46262025-03-08T00:50:30.847ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
46272025-03-08T00:50:30.847ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dfdc153-e3ae-496e-b3cc-241241ec31ba", "content-length": "109", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46282025-03-08T00:50:30.847ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
46292025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "185cba28-22c4-4581-bb1b-1625c9646a55", "content-length": "109", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46302025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
46312025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
46322025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46332025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31dc1496-85c8-458c-9796-02858d528e63", "content-length": "169", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46342025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
46352025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46362025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78a51790-1501-4f32-bd76-aa47b7ecef1d", "content-length": "169", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46372025-03-08T00:50:30.848ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
46382025-03-08T00:50:30.849ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd0f23d9-edf2-4e99-a600-3213a01bc822", "content-length": "178", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46392025-03-08T00:50:30.849ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
46402025-03-08T00:50:30.852ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d39f9d94-d704-4cb6-82c6-dac88d51effc", "content-length": "178", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46412025-03-08T00:50:30.852ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46422025-03-08T00:50:30.853ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
46432025-03-08T00:50:30.853ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46442025-03-08T00:50:30.853ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dcf5f8f-156b-4f6c-95a1-ca2455004f5b", "content-length": "702", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46452025-03-08T00:50:30.853ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
46462025-03-08T00:50:30.853ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c88f7549-5235-4472-9ce9-d59782e902e0", "content-length": "727", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46472025-03-08T00:50:30.853ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
46482025-03-08T00:50:30.854ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87ce56ef-9fb2-468f-a2a3-aecdf8c9b2cc", "content-length": "729", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46492025-03-08T00:50:30.854ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
46502025-03-08T00:50:30.854ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25ebc4bc-943f-4177-87d4-a5ec00f58844", "content-length": "728", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46512025-03-08T00:50:30.854ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46522025-03-08T00:50:30.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46532025-03-08T00:50:30.858ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46542025-03-08T00:50:30.858ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46552025-03-08T00:50:30.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46562025-03-08T00:50:30.860ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46572025-03-08T00:50:30.860ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46582025-03-08T00:50:30.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46592025-03-08T00:50:30.862ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46602025-03-08T00:50:30.862ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46612025-03-08T00:50:30.930ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
46622025-03-08T00:50:30.930ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "08a150f0-6960-4e12-a298-2b6e017611e3", "content-length": "691", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46632025-03-08T00:50:30.930ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
46642025-03-08T00:50:30.930ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
46652025-03-08T00:50:30.930ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
46662025-03-08T00:50:30.931ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46672025-03-08T00:50:30.932ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 639.561231ms iteration = 21 status = {}
46682025-03-08T00:50:30.932ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46692025-03-08T00:50:30.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46702025-03-08T00:50:30.933ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46712025-03-08T00:50:30.933ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46722025-03-08T00:50:30.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46732025-03-08T00:50:30.934ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46742025-03-08T00:50:30.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46752025-03-08T00:50:30.934ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46762025-03-08T00:50:30.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46772025-03-08T00:50:30.936ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46782025-03-08T00:50:30.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46792025-03-08T00:50:30.937ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
46802025-03-08T00:50:30.937ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46812025-03-08T00:50:30.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46822025-03-08T00:50:30.937ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46832025-03-08T00:50:30.942ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
46842025-03-08T00:50:30.942ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
46852025-03-08T00:50:30.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
46862025-03-08T00:50:30.947ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
46872025-03-08T00:50:30.948ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fbf1b68-bc21-472e-a663-f2819508c8c5", "content-length": "134", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
46882025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
46892025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
46902025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
46912025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
46922025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46932025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46942025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46952025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
46962025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
46972025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
46982025-03-08T00:50:30.948ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
46992025-03-08T00:50:30.948ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47002025-03-08T00:50:30.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47012025-03-08T00:50:30.970ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
47022025-03-08T00:50:30.970ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
47032025-03-08T00:50:30.972ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47042025-03-08T00:50:30.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47052025-03-08T00:50:30.973ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
47062025-03-08T00:50:30.973ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6a59941-bd4f-44d1-bf52-dbe2ae8e8a64", "content-length": "2", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
47072025-03-08T00:50:30.973ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
47082025-03-08T00:50:30.973ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a812639-8291-486a-8dc6-3bdf95574d15", "content-length": "52", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
47092025-03-08T00:50:30.973ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
47102025-03-08T00:50:30.973ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
47112025-03-08T00:50:30.974ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47122025-03-08T00:50:30.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47132025-03-08T00:50:30.977ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47142025-03-08T00:50:30.977ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47152025-03-08T00:50:30.980ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47162025-03-08T00:50:30.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47172025-03-08T00:50:30.981ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47182025-03-08T00:50:30.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47192025-03-08T00:50:30.983ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47202025-03-08T00:50:30.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47212025-03-08T00:50:31.014ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47222025-03-08T00:50:31.017ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
47232025-03-08T00:50:31.018ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47242025-03-08T00:50:31.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47252025-03-08T00:50:31.019ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47262025-03-08T00:50:31.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47272025-03-08T00:50:31.021ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47282025-03-08T00:50:31.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47292025-03-08T00:50:31.023ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47302025-03-08T00:50:31.028ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
47312025-03-08T00:50:31.028ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47322025-03-08T00:50:31.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47332025-03-08T00:50:31.032ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47342025-03-08T00:50:31.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47352025-03-08T00:50:31.033ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47362025-03-08T00:50:31.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47372025-03-08T00:50:31.035ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47382025-03-08T00:50:31.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47392025-03-08T00:50:31.037ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47402025-03-08T00:50:31.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47412025-03-08T00:50:31.040ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47422025-03-08T00:50:31.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47432025-03-08T00:50:31.041ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47442025-03-08T00:50:31.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47452025-03-08T00:50:31.043ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47462025-03-08T00:50:31.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47472025-03-08T00:50:31.044ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47482025-03-08T00:50:31.047ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
47492025-03-08T00:50:31.047ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47502025-03-08T00:50:31.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47512025-03-08T00:50:31.049ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47522025-03-08T00:50:31.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47532025-03-08T00:50:31.050ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47542025-03-08T00:50:31.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47552025-03-08T00:50:31.052ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47562025-03-08T00:50:31.055ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
47572025-03-08T00:50:31.055ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47582025-03-08T00:50:31.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47592025-03-08T00:50:31.063ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 4d0825be-9769-464b-9eb9-39c5eebe5d62, kind: Service { id: da0d8428-046e-431d-9a82-470b80ca232a }, name: Name("external-dns-da0d8428-046e-431d-9a82-470b80ca232a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 176])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d157e977-c32e-4a22-a13f-9a7473eb90e5, kind: Service { id: 59287b7e-0d61-4586-b7c5-e79476927737 }, name: Name("external-dns-59287b7e-0d61-4586-b7c5-e79476927737"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 101])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 0c3d6881-8d48-4d96-9ed0-85d27344279b, kind: Service { id: 4d603338-4154-45db-a26b-b6679dc7f0ec }, name: Name("nexus-4d603338-4154-45db-a26b-b6679dc7f0ec"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 106])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 3723c067-03a8-4ed5-9a97-5c3ee10bca5c, kind: Service { id: 841ee309-4e1e-443f-ae06-2ad5eb454a8d }, name: Name("nexus-841ee309-4e1e-443f-ae06-2ad5eb454a8d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 133])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d79ffed2-490d-44d6-943a-e4d940d7938a, kind: Service { id: 8b94fb8b-b81f-40e3-a7d8-ae559205ae7f }, name: Name("nexus-8b94fb8b-b81f-40e3-a7d8-ae559205ae7f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 217])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
47602025-03-08T00:50:31.063ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47612025-03-08T00:50:31.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47622025-03-08T00:50:31.064ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47632025-03-08T00:50:31.066ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
47642025-03-08T00:50:31.066ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47652025-03-08T00:50:31.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47662025-03-08T00:50:31.068ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
47672025-03-08T00:50:31.068ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4d0825be-9769-464b-9eb9-39c5eebe5d62, kind: Service { id: da0d8428-046e-431d-9a82-470b80ca232a }, name: Name("external-dns-da0d8428-046e-431d-9a82-470b80ca232a"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 176])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d157e977-c32e-4a22-a13f-9a7473eb90e5, kind: Service { id: 59287b7e-0d61-4586-b7c5-e79476927737 }, name: Name("external-dns-59287b7e-0d61-4586-b7c5-e79476927737"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 149, 101])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 0c3d6881-8d48-4d96-9ed0-85d27344279b, kind: Service { id: 4d603338-4154-45db-a26b-b6679dc7f0ec }, name: Name("nexus-4d603338-4154-45db-a26b-b6679dc7f0ec"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 106])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 3723c067-03a8-4ed5-9a97-5c3ee10bca5c, kind: Service { id: 841ee309-4e1e-443f-ae06-2ad5eb454a8d }, name: Name("nexus-841ee309-4e1e-443f-ae06-2ad5eb454a8d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 237, 133])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: d79ffed2-490d-44d6-943a-e4d940d7938a, kind: Service { id: 8b94fb8b-b81f-40e3-a7d8-ae559205ae7f }, name: Name("nexus-8b94fb8b-b81f-40e3-a7d8-ae559205ae7f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 217])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
47682025-03-08T00:50:31.068ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
47692025-03-08T00:50:31.102ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fa25a8ba-fe01-4f6e-a0f0-55bab6ac3e49", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
47702025-03-08T00:50:31.102ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
47712025-03-08T00:50:31.102ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
47722025-03-08T00:50:31.107ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf9d0ad8-55a5-4679-b40e-4f109ea8d5eb", "content-length": "682", "date": "Sat, 08 Mar 2025 00:50:31 GMT"} })
47732025-03-08T00:50:31.107ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47742025-03-08T00:50:31.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47752025-03-08T00:50:31.114ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47762025-03-08T00:50:31.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47772025-03-08T00:50:31.119ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47782025-03-08T00:50:31.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47792025-03-08T00:50:31.121ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47802025-03-08T00:50:31.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47812025-03-08T00:50:31.129ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 837.220248ms iteration = 21 status = {}
47822025-03-08T00:50:31.130ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 87826585-0843-40d2-a3ce-382fa54dbc5c, time_created: 2025-03-08T00:40:28.174204Z, time_modified: 2025-03-08T00:40:28.174204Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
47832025-03-08T00:50:31.130ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
47842025-03-08T00:50:31.130ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47852025-03-08T00:50:31.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47862025-03-08T00:50:31.133ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
47872025-03-08T00:50:31.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
47882025-03-08T00:50:31.135ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47892025-03-08T00:50:31.138ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
47902025-03-08T00:50:31.169ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
47912025-03-08T00:50:31.174ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "b2c1e972-9414-40c6-aff6-754869223bba", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
47922025-03-08T00:50:31.175ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
47932025-03-08T00:50:31.175ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 883.458759ms
47942025-03-08T00:50:31.175ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 883.468619ms iteration = 3 status = {}
47952025-03-08T00:50:31.231ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "40f5acbf-d870-4503-9ad3-56b31b028fdb", "content-length": "31970", "date": "Sat, 08 Mar 2025 00:50:30 GMT"} })
47962025-03-08T00:50:31.233ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
47972025-03-08T00:50:31.233ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
47982025-03-08T00:50:31.233ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
47992025-03-08T00:50:31.235ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48002025-03-08T00:50:31.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48012025-03-08T00:50:31.237ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48022025-03-08T00:50:31.239ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
48032025-03-08T00:50:31.239ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48042025-03-08T00:50:31.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48052025-03-08T00:50:31.434ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 8c5284e2-0e05-44ac-8271-2abaacd44cef file = nexus/db-queries/src/db/datastore/inventory.rs:1015
48062025-03-08T00:50:31.434ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 8c5284e2-0e05-44ac-8271-2abaacd44cef time_started = 2025-03-08 00:50:30.841999 UTC
48072025-03-08T00:50:31.434ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.142389088s iteration = 2 status = {"collection_id":"8c5284e2-0e05-44ac-8271-2abaacd44cef","time_done":"2025-03-08 00:50:31.233295 UTC","time_started":"2025-03-08 00:50:30.841999 UTC"}
48082025-03-08T00:50:31.434ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
48092025-03-08T00:50:31.434ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
48102025-03-08T00:50:31.434ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48112025-03-08T00:50:31.434ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48122025-03-08T00:50:31.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48132025-03-08T00:50:31.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48142025-03-08T00:50:31.437ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48152025-03-08T00:50:31.437ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48162025-03-08T00:50:31.440ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
48172025-03-08T00:50:31.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48182025-03-08T00:50:31.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48192025-03-08T00:50:31.440ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
48202025-03-08T00:50:31.440ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48212025-03-08T00:50:31.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48222025-03-08T00:50:31.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48232025-03-08T00:50:31.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48242025-03-08T00:50:31.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48252025-03-08T00:50:31.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48262025-03-08T00:50:31.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48272025-03-08T00:50:31.447ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
48282025-03-08T00:50:31.447ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48292025-03-08T00:50:31.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4830 Batch: [InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 8c5284e2-0e05-44ac-8271-2abaacd44cef (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
48312025-03-08T00:50:31.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48322025-03-08T00:50:31.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48332025-03-08T00:50:31.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48342025-03-08T00:50:31.478ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
48352025-03-08T00:50:31.478ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48362025-03-08T00:50:31.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48372025-03-08T00:50:31.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48382025-03-08T00:50:31.482ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
48392025-03-08T00:50:31.482ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48402025-03-08T00:50:31.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48412025-03-08T00:50:31.484ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48422025-03-08T00:50:31.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48432025-03-08T00:50:31.486ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48442025-03-08T00:50:31.488ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
48452025-03-08T00:50:31.488ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48462025-03-08T00:50:31.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48472025-03-08T00:50:31.489ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 54.445323ms iteration = 23 status = {"physical_disks_added":0}
48482025-03-08T00:50:31.490ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48492025-03-08T00:50:31.492ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
48502025-03-08T00:50:31.492ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
48512025-03-08T00:50:31.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
48522025-03-08T00:50:31.495ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
48532025-03-08T00:50:31.495ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 60.225956ms iteration = 5 status = {"blueprint_id":"a346f089-8033-4b92-b854-361d1da5cab2","inventory_collection_id":"8c5284e2-0e05-44ac-8271-2abaacd44cef","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
48542025-03-08T00:50:32.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 379 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 91023579-34f8-499b-94ff-cd2b7b2c882f response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
48552025-03-08T00:50:40.292ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
48562025-03-08T00:50:40.292ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48572025-03-08T00:50:40.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48582025-03-08T00:50:40.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
48592025-03-08T00:50:40.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48602025-03-08T00:50:40.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48612025-03-08T00:50:40.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48622025-03-08T00:50:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48632025-03-08T00:50:40.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48642025-03-08T00:50:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48652025-03-08T00:50:40.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48662025-03-08T00:50:40.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48672025-03-08T00:50:40.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48682025-03-08T00:50:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
48692025-03-08T00:50:40.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48702025-03-08T00:50:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48712025-03-08T00:50:40.318ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
48722025-03-08T00:50:40.318ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.044043ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
48732025-03-08T00:50:40.322ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48742025-03-08T00:50:40.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48752025-03-08T00:50:40.353ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.022595ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
48762025-03-08T00:50:42.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 307 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 86bd4488-e15f-498d-955c-20931b644d89 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
48772025-03-08T00:50:50.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
48782025-03-08T00:50:50.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48792025-03-08T00:50:50.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48802025-03-08T00:50:50.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
48812025-03-08T00:50:50.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48822025-03-08T00:50:50.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48832025-03-08T00:50:50.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48842025-03-08T00:50:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48852025-03-08T00:50:50.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48862025-03-08T00:50:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48872025-03-08T00:50:50.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48882025-03-08T00:50:50.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48892025-03-08T00:50:50.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48902025-03-08T00:50:50.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
48912025-03-08T00:50:50.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48922025-03-08T00:50:50.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48932025-03-08T00:50:50.315ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
48942025-03-08T00:50:50.315ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.770837ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
48952025-03-08T00:50:50.328ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48962025-03-08T00:50:50.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48972025-03-08T00:50:50.330ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.232704ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
48982025-03-08T00:50:52.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 327 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 94825cc8-f539-422d-a253-6b7b196bc1aa response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
48992025-03-08T00:50:57.638ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:38739
49002025-03-08T00:50:57.639ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:38739 req_id = 948ec9c0-6ccc-42dd-ac9e-9150cef25e25 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
49012025-03-08T00:50:57.639ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:38739 req_id = 948ec9c0-6ccc-42dd-ac9e-9150cef25e25 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
49022025-03-08T00:50:57.644ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 5351 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:38739 req_id = 948ec9c0-6ccc-42dd-ac9e-9150cef25e25 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
49032025-03-08T00:50:57.644ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:38739 req_id = 6b1e31a8-2509-4b84-9a0d-a8eca9c82ad7 roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
49042025-03-08T00:50:57.645ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:38739 req_id = 6b1e31a8-2509-4b84-9a0d-a8eca9c82ad7 resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
49052025-03-08T00:50:57.648ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3736 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:38739 req_id = 6b1e31a8-2509-4b84-9a0d-a8eca9c82ad7 response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
49062025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
49072025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
49082025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
49092025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
49102025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
49112025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
49122025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49132025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49142025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
49152025-03-08T00:51:00.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
49162025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
49172025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
49182025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
49192025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
49202025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49212025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
49222025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
49232025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
49242025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
49252025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
49262025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
49272025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
49282025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
49292025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
49302025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
49312025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
49322025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49332025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
49342025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49352025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
49362025-03-08T00:51:00.292ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
49372025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
49382025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49392025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49402025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
49412025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49422025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
49432025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49442025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
49452025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49462025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
49472025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49482025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49492025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
49502025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
49512025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
49522025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
49532025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
49542025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49552025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
49562025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49572025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49582025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49592025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49602025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49612025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49622025-03-08T00:51:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49632025-03-08T00:51:00.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49642025-03-08T00:51:00.294ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1543, tv_nsec: 318468954 } })
49652025-03-08T00:51:00.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49662025-03-08T00:51:00.294ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
49672025-03-08T00:51:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1543, tv_nsec: 319199312 } })
49682025-03-08T00:51:00.295ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
49692025-03-08T00:51:00.295ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.625794ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
49702025-03-08T00:51:00.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49712025-03-08T00:51:00.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49722025-03-08T00:51:00.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49732025-03-08T00:51:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
49742025-03-08T00:51:00.298ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49752025-03-08T00:51:00.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49762025-03-08T00:51:00.300ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49772025-03-08T00:51:00.302ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
49782025-03-08T00:51:00.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
49792025-03-08T00:51:00.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
49802025-03-08T00:51:00.303ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49812025-03-08T00:51:00.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49822025-03-08T00:51:00.305ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49832025-03-08T00:51:00.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
49842025-03-08T00:51:00.309ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49852025-03-08T00:51:00.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49862025-03-08T00:51:00.310ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49872025-03-08T00:51:00.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49882025-03-08T00:51:00.315ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49892025-03-08T00:51:00.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49902025-03-08T00:51:00.323ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1543, tv_nsec: 347136692 } })
49912025-03-08T00:51:00.323ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
4992 Batch: [InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
49932025-03-08T00:51:00.355ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
49942025-03-08T00:51:00.355ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49952025-03-08T00:51:00.356ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccb36dee-76b8-423a-816d-3dfe83c3005b", "content-length": "26", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
49962025-03-08T00:51:00.356ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49972025-03-08T00:51:00.356ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 64.077651ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
49982025-03-08T00:51:00.357ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
49992025-03-08T00:51:00.358ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50002025-03-08T00:51:00.358ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
50012025-03-08T00:51:00.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
50022025-03-08T00:51:00.362ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50032025-03-08T00:51:00.362ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50042025-03-08T00:51:00.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50052025-03-08T00:51:00.363ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50062025-03-08T00:51:00.363ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
50072025-03-08T00:51:00.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50082025-03-08T00:51:00.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50092025-03-08T00:51:00.364ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50102025-03-08T00:51:00.364ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
50112025-03-08T00:51:00.364ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
50122025-03-08T00:51:00.364ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "80ee530d-8352-43aa-adc9-8486378e54ec", "content-length": "26", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
50132025-03-08T00:51:00.364ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
50142025-03-08T00:51:00.364ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 72.749535ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
50152025-03-08T00:51:00.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50162025-03-08T00:51:00.365ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50172025-03-08T00:51:00.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50182025-03-08T00:51:00.366ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50192025-03-08T00:51:00.367ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
50202025-03-08T00:51:00.367ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 75.329359ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
50212025-03-08T00:51:00.367ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50222025-03-08T00:51:00.368ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50232025-03-08T00:51:00.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50242025-03-08T00:51:00.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
50252025-03-08T00:51:00.370ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
50262025-03-08T00:51:00.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
50272025-03-08T00:51:00.370ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50282025-03-08T00:51:00.371ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.896807ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
50292025-03-08T00:51:00.371ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
50302025-03-08T00:51:00.371ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50312025-03-08T00:51:00.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50322025-03-08T00:51:00.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50332025-03-08T00:51:00.374ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50342025-03-08T00:51:00.374ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50352025-03-08T00:51:00.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50362025-03-08T00:51:00.376ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
50372025-03-08T00:51:00.376ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50382025-03-08T00:51:00.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50392025-03-08T00:51:00.379ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50402025-03-08T00:51:00.380ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 88.056183ms iteration = 22 status = 0
50412025-03-08T00:51:00.380ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50422025-03-08T00:51:00.380ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
50432025-03-08T00:51:00.380ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
50442025-03-08T00:51:00.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
50452025-03-08T00:51:00.382ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50462025-03-08T00:51:00.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50472025-03-08T00:51:00.382ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50482025-03-08T00:51:00.382ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50492025-03-08T00:51:00.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50502025-03-08T00:51:00.384ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50512025-03-08T00:51:00.387ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
50522025-03-08T00:51:00.387ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50532025-03-08T00:51:00.387ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50542025-03-08T00:51:00.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50552025-03-08T00:51:00.394ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
50562025-03-08T00:51:00.394ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50572025-03-08T00:51:00.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50582025-03-08T00:51:00.403ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 111.182374ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
50592025-03-08T00:51:00.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
50602025-03-08T00:51:00.414ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 122.236456ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
50612025-03-08T00:51:00.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50622025-03-08T00:51:00.430ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50632025-03-08T00:51:00.430ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
50642025-03-08T00:51:00.430ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
50652025-03-08T00:51:00.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
50662025-03-08T00:51:00.462ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
50672025-03-08T00:51:00.462ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
50682025-03-08T00:51:00.462ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2519f566-d1cb-4608-9411-de668295614c", "content-length": "26", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
50692025-03-08T00:51:00.462ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
50702025-03-08T00:51:00.464ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
50712025-03-08T00:51:00.470ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50722025-03-08T00:51:00.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50732025-03-08T00:51:00.488ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 196.54121ms iteration = 24 status = {"physical_disks_added":0}
50742025-03-08T00:51:00.493ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
50752025-03-08T00:51:00.493ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "385e5ab2-6fa6-46c1-b7ee-2dd899222d77", "content-length": "1", "date": "Sat, 08 Mar 2025 00:50:59 GMT"} })
50762025-03-08T00:51:00.494ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
50772025-03-08T00:51:00.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
50782025-03-08T00:51:00.521ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
50792025-03-08T00:51:00.521ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
50802025-03-08T00:51:00.521ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "731d7949-45b3-42c3-8334-2cc4db8a8f74", "content-length": "525", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
50812025-03-08T00:51:00.521ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50822025-03-08T00:51:00.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50832025-03-08T00:51:00.525ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/de771ab5-f863-4317-9c0b-66d2cc459f21/state
50842025-03-08T00:51:00.527ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 236.805953ms iteration = 22 status = null
50852025-03-08T00:51:00.531ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 240.207567ms iteration = 22 status = {}
50862025-03-08T00:51:00.537ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
50872025-03-08T00:51:00.537ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 246.7809ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
50882025-03-08T00:51:00.558ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 266.168148ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
50892025-03-08T00:51:00.559ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
50902025-03-08T00:51:00.559ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
50912025-03-08T00:51:00.559ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 267.434309ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
50922025-03-08T00:51:00.560ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
50932025-03-08T00:51:00.560ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 268.426044ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
50942025-03-08T00:51:00.576ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {efd04917-3123-4423-a590-74a40f32a6ed: {10.151.2.175: {69e9d00b-ef75-41a7-ad47-501e75752c8d}, 10.151.2.176: {69e9d00b-ef75-41a7-ad47-501e75752c8d}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
50952025-03-08T00:51:00.576ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
50962025-03-08T00:51:00.577ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c2f0fad9-dbad-4406-a962-e7841ee29326", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
50972025-03-08T00:51:00.577ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50982025-03-08T00:51:00.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50992025-03-08T00:51:00.579ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51002025-03-08T00:51:00.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51012025-03-08T00:51:00.582ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51022025-03-08T00:51:00.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51032025-03-08T00:51:00.586ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51042025-03-08T00:51:00.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51052025-03-08T00:51:00.588ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51062025-03-08T00:51:00.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51072025-03-08T00:51:00.596ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51082025-03-08T00:51:00.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51092025-03-08T00:51:00.601ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
51102025-03-08T00:51:00.601ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51112025-03-08T00:51:00.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51122025-03-08T00:51:00.604ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51132025-03-08T00:51:00.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51142025-03-08T00:51:00.606ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51152025-03-08T00:51:00.607ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51162025-03-08T00:51:00.609ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51172025-03-08T00:51:00.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51182025-03-08T00:51:00.611ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51192025-03-08T00:51:00.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51202025-03-08T00:51:00.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51212025-03-08T00:51:00.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51222025-03-08T00:51:00.615ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51232025-03-08T00:51:00.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51242025-03-08T00:51:00.620ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51252025-03-08T00:51:00.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51262025-03-08T00:51:00.623ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
51272025-03-08T00:51:00.623ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51282025-03-08T00:51:00.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51292025-03-08T00:51:00.634ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
51302025-03-08T00:51:00.635ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f796cb1-9e41-4395-bf92-aec83d17c795", "content-length": "134", "date": "Sat, 08 Mar 2025 00:50:59 GMT"} })
51312025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
51322025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
51332025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
51342025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
51352025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51362025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51372025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51382025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
51392025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
51402025-03-08T00:51:00.635ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
51412025-03-08T00:51:00.636ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
51422025-03-08T00:51:00.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51432025-03-08T00:51:00.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51442025-03-08T00:51:00.637ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
51452025-03-08T00:51:00.638ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "87223122-f4e2-445b-815c-75f5c6ae5b81", "content-length": "2", "date": "Sat, 08 Mar 2025 00:50:59 GMT"} })
51462025-03-08T00:51:00.638ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
51472025-03-08T00:51:00.638ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51482025-03-08T00:51:00.638ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "96941c3d-31c3-424a-ac84-7b72c189d8fa", "content-length": "52", "date": "Sat, 08 Mar 2025 00:50:59 GMT"} })
51492025-03-08T00:51:00.638ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
51502025-03-08T00:51:00.638ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
51512025-03-08T00:51:00.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51522025-03-08T00:51:00.642ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51532025-03-08T00:51:00.646ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
51542025-03-08T00:51:00.646ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51552025-03-08T00:51:00.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51562025-03-08T00:51:00.666ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51572025-03-08T00:51:00.672ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 5903a422-2123-4b6a-9e2a-1da7d551206b, lookup_type: ById(5903a422-2123-4b6a-9e2a-1da7d551206b) }, key: 3b59e42e-8aa7-499a-a283-a107ad23826a, lookup_type: ById(3b59e42e-8aa7-499a-a283-a107ad23826a) }, key: 83dff6c3-0217-4f04-a11e-9620947b6ec0, lookup_type: ById(83dff6c3-0217-4f04-a11e-9620947b6ec0) }, key: 9f763912-569c-447c-ab9d-133c00d90394, lookup_type: ById(9f763912-569c-447c-ab9d-133c00d90394) } result = Ok(())
51582025-03-08T00:51:00.672ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51592025-03-08T00:51:00.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51602025-03-08T00:51:00.697ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51612025-03-08T00:51:00.699ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
51622025-03-08T00:51:00.699ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51632025-03-08T00:51:00.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51642025-03-08T00:51:00.710ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51652025-03-08T00:51:00.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51662025-03-08T00:51:00.712ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51672025-03-08T00:51:00.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51682025-03-08T00:51:00.714ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51692025-03-08T00:51:00.719ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 5903a422-2123-4b6a-9e2a-1da7d551206b, lookup_type: ById(5903a422-2123-4b6a-9e2a-1da7d551206b) }, key: 3b59e42e-8aa7-499a-a283-a107ad23826a, lookup_type: ById(3b59e42e-8aa7-499a-a283-a107ad23826a) }, key: 83dff6c3-0217-4f04-a11e-9620947b6ec0, lookup_type: ById(83dff6c3-0217-4f04-a11e-9620947b6ec0) }, key: 9f763912-569c-447c-ab9d-133c00d90394, lookup_type: ById(9f763912-569c-447c-ab9d-133c00d90394) } result = Ok(())
51702025-03-08T00:51:00.719ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51712025-03-08T00:51:00.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51722025-03-08T00:51:00.723ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51732025-03-08T00:51:00.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51742025-03-08T00:51:00.728ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51752025-03-08T00:51:00.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51762025-03-08T00:51:00.730ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51772025-03-08T00:51:00.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51782025-03-08T00:51:00.733ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51792025-03-08T00:51:00.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51802025-03-08T00:51:00.735ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51812025-03-08T00:51:00.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51822025-03-08T00:51:00.737ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51832025-03-08T00:51:00.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51842025-03-08T00:51:00.739ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51852025-03-08T00:51:00.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51862025-03-08T00:51:00.777ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5eafd6d8-fb6e-427f-9506-c26220e24e9d", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
51872025-03-08T00:51:00.777ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
51882025-03-08T00:51:00.777ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
51892025-03-08T00:51:00.778ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6560ec91-10b6-471e-b3dc-21119cb71b03", "content-length": "682", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
51902025-03-08T00:51:00.778ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51912025-03-08T00:51:00.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51922025-03-08T00:51:00.782ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51932025-03-08T00:51:00.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51942025-03-08T00:51:00.786ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51952025-03-08T00:51:00.787ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51962025-03-08T00:51:00.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51972025-03-08T00:51:00.789ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51982025-03-08T00:51:00.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51992025-03-08T00:51:00.790ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 5903a422-2123-4b6a-9e2a-1da7d551206b, lookup_type: ById(5903a422-2123-4b6a-9e2a-1da7d551206b) }, key: 3b59e42e-8aa7-499a-a283-a107ad23826a, lookup_type: ById(3b59e42e-8aa7-499a-a283-a107ad23826a) }, key: 83dff6c3-0217-4f04-a11e-9620947b6ec0, lookup_type: ById(83dff6c3-0217-4f04-a11e-9620947b6ec0) }, key: 909816e7-c5f5-43dc-bcec-437e5ab97e9a, lookup_type: ById(909816e7-c5f5-43dc-bcec-437e5ab97e9a) } result = Ok(())
52002025-03-08T00:51:00.790ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52012025-03-08T00:51:00.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52022025-03-08T00:51:00.796ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 504.277283ms iteration = 22 status = {}
52032025-03-08T00:51:00.808ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
52042025-03-08T00:51:00.809ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f59da18-4d2b-45a8-8819-b1c9514dd332", "content-length": "811", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
52052025-03-08T00:51:00.809ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(13441751) }} sled = sock
52062025-03-08T00:51:00.809ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
52072025-03-08T00:51:00.809ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
52082025-03-08T00:51:00.810ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52092025-03-08T00:51:00.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52102025-03-08T00:51:00.811ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 518.825152ms iteration = 22 status = {}
52112025-03-08T00:51:00.812ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52122025-03-08T00:51:00.813ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52132025-03-08T00:51:00.814ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52142025-03-08T00:51:00.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52152025-03-08T00:51:00.818ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52162025-03-08T00:51:00.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52172025-03-08T00:51:00.847ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52182025-03-08T00:51:00.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52192025-03-08T00:51:00.880ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52202025-03-08T00:51:00.881ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52212025-03-08T00:51:00.920ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52222025-03-08T00:51:00.923ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 5903a422-2123-4b6a-9e2a-1da7d551206b, lookup_type: ById(5903a422-2123-4b6a-9e2a-1da7d551206b) }, key: 3b59e42e-8aa7-499a-a283-a107ad23826a, lookup_type: ById(3b59e42e-8aa7-499a-a283-a107ad23826a) }, key: 83dff6c3-0217-4f04-a11e-9620947b6ec0, lookup_type: ById(83dff6c3-0217-4f04-a11e-9620947b6ec0) }, key: 909816e7-c5f5-43dc-bcec-437e5ab97e9a, lookup_type: ByName("default") } result = Ok(())
52232025-03-08T00:51:00.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52242025-03-08T00:51:00.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52252025-03-08T00:51:00.926ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52262025-03-08T00:51:00.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52272025-03-08T00:51:00.928ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52282025-03-08T00:51:00.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52292025-03-08T00:51:00.931ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52302025-03-08T00:51:00.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52312025-03-08T00:51:00.933ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52322025-03-08T00:51:00.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52332025-03-08T00:51:00.934ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52342025-03-08T00:51:00.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52352025-03-08T00:51:00.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52362025-03-08T00:51:00.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52372025-03-08T00:51:00.937ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52382025-03-08T00:51:00.941ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = InternetGateway { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 5903a422-2123-4b6a-9e2a-1da7d551206b, lookup_type: ById(5903a422-2123-4b6a-9e2a-1da7d551206b) }, key: 3b59e42e-8aa7-499a-a283-a107ad23826a, lookup_type: ById(3b59e42e-8aa7-499a-a283-a107ad23826a) }, key: 83dff6c3-0217-4f04-a11e-9620947b6ec0, lookup_type: ById(83dff6c3-0217-4f04-a11e-9620947b6ec0) }, key: 69e9d00b-ef75-41a7-ad47-501e75752c8d, lookup_type: ByName("default") } result = Ok(())
52392025-03-08T00:51:00.941ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
52402025-03-08T00:51:00.942ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4a58a6ec-ecd6-46a9-9722-e6777a8f27a9", "date": "Sat, 08 Mar 2025 00:51:00 GMT"} })
52412025-03-08T00:51:00.943ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 651.443474ms iteration = 22 status = {}
52422025-03-08T00:51:02.650ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 329 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = a58914a5-4476-4353-b404-3e0719838377 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
52432025-03-08T00:51:02.753ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/de771ab5-f863-4317-9c0b-66d2cc459f21/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "6730d941-8032-41a4-a772-47feca0053fd", "content-length": "131", "date": "Sat, 08 Mar 2025 00:51:02 GMT"} })
52442025-03-08T00:51:02.763ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 6c0adf12-3120-40cb-8319-60c76e95db9d sled_id = 87826585-0843-40d2-a3ce-382fa54dbc5c state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2025-03-08T00:50:54.850840242Z }, migration_in: None, migration_out: None } vmm_id = de771ab5-f863-4317-9c0b-66d2cc459f21
52452025-03-08T00:51:02.763ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2088 instance_id = 6c0adf12-3120-40cb-8319-60c76e95db9d migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = de771ab5-f863-4317-9c0b-66d2cc459f21 sled_id = 87826585-0843-40d2-a3ce-382fa54dbc5c vmm_id = de771ab5-f863-4317-9c0b-66d2cc459f21 vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2025-03-08T00:50:54.850840242Z }
52462025-03-08T00:51:02.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 6c0adf12-3120-40cb-8319-60c76e95db9d roles = RoleSet { roles: {} } sled_id = 87826585-0843-40d2-a3ce-382fa54dbc5c vmm_id = de771ab5-f863-4317-9c0b-66d2cc459f21
52472025-03-08T00:51:02.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 6c0adf12-3120-40cb-8319-60c76e95db9d resource = Database result = Ok(()) sled_id = 87826585-0843-40d2-a3ce-382fa54dbc5c vmm_id = de771ab5-f863-4317-9c0b-66d2cc459f21
52482025-03-08T00:51:02.813ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
52492025-03-08T00:51:02.814ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 2.522670715s iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"starting":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
52502025-03-08T00:51:10.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
52512025-03-08T00:51:10.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52522025-03-08T00:51:10.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52532025-03-08T00:51:10.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
52542025-03-08T00:51:10.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52552025-03-08T00:51:10.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52562025-03-08T00:51:10.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52572025-03-08T00:51:10.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
52582025-03-08T00:51:10.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52592025-03-08T00:51:10.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52602025-03-08T00:51:10.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52612025-03-08T00:51:10.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52622025-03-08T00:51:10.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52632025-03-08T00:51:10.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
52642025-03-08T00:51:10.305ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52652025-03-08T00:51:10.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52662025-03-08T00:51:10.327ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
52672025-03-08T00:51:10.327ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 35.450015ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
52682025-03-08T00:51:10.327ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52692025-03-08T00:51:10.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52702025-03-08T00:51:10.363ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.596102ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
52712025-03-08T00:51:12.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 458 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 44941770-5fb4-4a5b-9def-a12d8b45b412 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
52722025-03-08T00:51:20.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
52732025-03-08T00:51:20.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52742025-03-08T00:51:20.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52752025-03-08T00:51:20.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
52762025-03-08T00:51:20.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52772025-03-08T00:51:20.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52782025-03-08T00:51:20.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52792025-03-08T00:51:20.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
52802025-03-08T00:51:20.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52812025-03-08T00:51:20.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52822025-03-08T00:51:20.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52832025-03-08T00:51:20.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52842025-03-08T00:51:20.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52852025-03-08T00:51:20.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
52862025-03-08T00:51:20.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52872025-03-08T00:51:20.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52882025-03-08T00:51:20.315ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
52892025-03-08T00:51:20.315ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.426803ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
52902025-03-08T00:51:20.322ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52912025-03-08T00:51:20.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52922025-03-08T00:51:20.324ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.450551ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
52932025-03-08T00:51:22.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 428 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 36cbdd3a-895a-4260-ad03-626949d1eb52 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
52942025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
52952025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
52962025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
52972025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
52982025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52992025-03-08T00:51:30.291ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-08 00:41:30.291385198 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
53002025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
53012025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
53022025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
53032025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
53042025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53052025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
53062025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
53072025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
53082025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
53092025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53102025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
53112025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
53122025-03-08T00:51:30.291ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = a346f089-8033-4b92-b854-361d1da5cab2
53132025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.43\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"a346f089-8033-4b92-b854-361d1da5cab2"}
53142025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
53152025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
53162025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
53172025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
53182025-03-08T00:51:30.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53192025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
53202025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53212025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
53222025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
53232025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
53242025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
53252025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
53262025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53272025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
53282025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
53292025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
53302025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53312025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
53322025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
53332025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
53342025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
53352025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
53362025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
53372025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
53382025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
53392025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
53402025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
53412025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
53422025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
53432025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53442025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
53452025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
53462025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
53472025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
53482025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53492025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
53502025-03-08T00:51:30.292ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
53512025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
53522025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
53532025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
53542025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
53552025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
53562025-03-08T00:51:30.292ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
53572025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53582025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53592025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
53602025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
53612025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
53622025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
53632025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
53642025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
53652025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
53662025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
53672025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53682025-03-08T00:51:30.292ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("59287b7e-0d61-4586-b7c5-e79476927737.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("da0d8428-046e-431d-9a82-470b80ca232a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1573, tv_nsec: 316956477 } })
53692025-03-08T00:51:30.292ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1ac98a83-1243-46f5-99db-fd29222026d8.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("508de01a-93ec-4309-a59e-0cfd22b5027a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("7a2ff4e8-6697-4c29-ad00-14a8b25b198c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1573, tv_nsec: 316965217 } })
53702025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
53712025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
53722025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
53732025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
53742025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
53752025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
53762025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
53772025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53782025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
53792025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
53802025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53812025-03-08T00:51:30.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
53822025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
53832025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
53842025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
53852025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
53862025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53872025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
53882025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
53892025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.680661ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
53902025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
53912025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.688822ms iteration = 12 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
53922025-03-08T00:51:30.293ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53932025-03-08T00:51:30.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53942025-03-08T00:51:30.294ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53952025-03-08T00:51:30.295ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.500451ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
53962025-03-08T00:51:30.296ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
53972025-03-08T00:51:30.296ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
53982025-03-08T00:51:30.296ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1573, tv_nsec: 320671859 } })
53992025-03-08T00:51:30.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54002025-03-08T00:51:30.296ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
54012025-03-08T00:51:30.297ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54022025-03-08T00:51:30.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54032025-03-08T00:51:30.298ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-08 00:41:30.291385198 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
54042025-03-08T00:51:30.298ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.997833ms iteration = 12 status = {"expiration":"2025-03-08T00:41:30.291385198Z","pruned":[]}
54052025-03-08T00:51:30.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54062025-03-08T00:51:30.299ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1573, tv_nsec: 323288633 } })
54072025-03-08T00:51:30.299ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
54082025-03-08T00:51:30.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
54092025-03-08T00:51:30.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54102025-03-08T00:51:30.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54112025-03-08T00:51:30.304ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54122025-03-08T00:51:30.305ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
54132025-03-08T00:51:30.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
54142025-03-08T00:51:30.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
54152025-03-08T00:51:30.307ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54162025-03-08T00:51:30.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54172025-03-08T00:51:30.310ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 17.943913ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
54182025-03-08T00:51:30.310ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54192025-03-08T00:51:30.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
54202025-03-08T00:51:30.313ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
54212025-03-08T00:51:30.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
54222025-03-08T00:51:30.319ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54232025-03-08T00:51:30.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54242025-03-08T00:51:30.324ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54252025-03-08T00:51:30.324ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54262025-03-08T00:51:30.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54272025-03-08T00:51:30.328ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
54282025-03-08T00:51:30.332ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
54292025-03-08T00:51:30.332ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 40.261024ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
54302025-03-08T00:51:30.336ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1573, tv_nsec: 360407979 } })
54312025-03-08T00:51:30.336ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
54322025-03-08T00:51:30.353ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 61.454457ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
54332025-03-08T00:51:30.357ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
54342025-03-08T00:51:30.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
54352025-03-08T00:51:30.358ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54362025-03-08T00:51:30.361ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54372025-03-08T00:51:30.362ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
54382025-03-08T00:51:30.362ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
54392025-03-08T00:51:30.362ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
54402025-03-08T00:51:30.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
54412025-03-08T00:51:30.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
54422025-03-08T00:51:30.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
54432025-03-08T00:51:30.364ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
54442025-03-08T00:51:30.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
54452025-03-08T00:51:30.367ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
54462025-03-08T00:51:30.367ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
54472025-03-08T00:51:30.367ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
54482025-03-08T00:51:30.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
54492025-03-08T00:51:30.369ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
54502025-03-08T00:51:30.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
54512025-03-08T00:51:30.373ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54522025-03-08T00:51:30.376ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
54532025-03-08T00:51:30.377ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
54542025-03-08T00:51:30.377ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 84.747041ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
54552025-03-08T00:51:30.377ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
54562025-03-08T00:51:30.377ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54572025-03-08T00:51:30.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
54582025-03-08T00:51:30.381ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 89.033717ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
54592025-03-08T00:51:30.382ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54602025-03-08T00:51:30.382ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54612025-03-08T00:51:30.382ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54622025-03-08T00:51:30.385ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54632025-03-08T00:51:30.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
54642025-03-08T00:51:30.386ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
54652025-03-08T00:51:30.386ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54662025-03-08T00:51:30.386ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
54672025-03-08T00:51:30.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54682025-03-08T00:51:30.386ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 94.064641ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
54692025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
54702025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
54712025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
54722025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
54732025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
54742025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
54752025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54762025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 94.915253ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
54772025-03-08T00:51:30.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54782025-03-08T00:51:30.390ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 98.536531ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
54792025-03-08T00:51:30.391ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
54802025-03-08T00:51:30.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
54812025-03-08T00:51:30.394ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
54822025-03-08T00:51:30.394ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
54832025-03-08T00:51:30.394ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
54842025-03-08T00:51:30.394ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54852025-03-08T00:51:30.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54862025-03-08T00:51:30.396ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5487 Batch: [InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
54882025-03-08T00:51:30.400ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
54892025-03-08T00:51:30.400ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54902025-03-08T00:51:30.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54912025-03-08T00:51:30.405ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
54922025-03-08T00:51:30.405ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
54932025-03-08T00:51:30.405ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 113.575533ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
54942025-03-08T00:51:30.410ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54952025-03-08T00:51:30.413ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
54962025-03-08T00:51:30.413ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54972025-03-08T00:51:30.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54982025-03-08T00:51:30.419ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
54992025-03-08T00:51:30.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55002025-03-08T00:51:30.422ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55012025-03-08T00:51:30.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55022025-03-08T00:51:30.424ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55032025-03-08T00:51:30.424ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55042025-03-08T00:51:30.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
55052025-03-08T00:51:30.426ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55062025-03-08T00:51:30.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55072025-03-08T00:51:30.429ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
55082025-03-08T00:51:30.429ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55092025-03-08T00:51:30.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55102025-03-08T00:51:30.430ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 138.151178ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
55112025-03-08T00:51:30.438ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55122025-03-08T00:51:30.438ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 145.967272ms iteration = 23 status = 0
55132025-03-08T00:51:30.441ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
55142025-03-08T00:51:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
55152025-03-08T00:51:30.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55162025-03-08T00:51:30.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55172025-03-08T00:51:30.444ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
55182025-03-08T00:51:30.444ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
55192025-03-08T00:51:30.444ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
55202025-03-08T00:51:30.444ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55212025-03-08T00:51:30.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55222025-03-08T00:51:30.444ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
55232025-03-08T00:51:30.446ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55242025-03-08T00:51:30.447ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 155.172149ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
55252025-03-08T00:51:30.449ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
55262025-03-08T00:51:30.449ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55272025-03-08T00:51:30.449ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
55282025-03-08T00:51:30.449ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
55292025-03-08T00:51:30.449ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
55302025-03-08T00:51:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55312025-03-08T00:51:30.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
55322025-03-08T00:51:30.451ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
55332025-03-08T00:51:30.451ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
55342025-03-08T00:51:30.451ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d49199-d2a1-42c0-a815-ce5920518124", "content-length": "26", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
55352025-03-08T00:51:30.451ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
55362025-03-08T00:51:30.452ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
55372025-03-08T00:51:30.452ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55382025-03-08T00:51:30.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
55392025-03-08T00:51:30.454ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
55402025-03-08T00:51:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
55412025-03-08T00:51:30.455ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55422025-03-08T00:51:30.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55432025-03-08T00:51:30.460ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
55442025-03-08T00:51:30.460ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 168.790735ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
55452025-03-08T00:51:30.460ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
55462025-03-08T00:51:30.460ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 0 version = 2
55472025-03-08T00:51:30.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
55482025-03-08T00:51:30.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
55492025-03-08T00:51:30.466ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55502025-03-08T00:51:30.468ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
55512025-03-08T00:51:30.468ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
55522025-03-08T00:51:30.469ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "11340111-2622-471e-8428-47784d5c863c", "content-length": "26", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
55532025-03-08T00:51:30.469ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
55542025-03-08T00:51:30.470ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
55552025-03-08T00:51:30.470ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
55562025-03-08T00:51:30.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
55572025-03-08T00:51:30.470ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
55582025-03-08T00:51:30.470ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
55592025-03-08T00:51:30.471ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "27c9e8ed-f209-4a8b-b18e-9f760c005eaf", "content-length": "26", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
55602025-03-08T00:51:30.471ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
55612025-03-08T00:51:30.471ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
55622025-03-08T00:51:30.472ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0a79fc16-6e48-4046-b57c-04fd4b1262a4 dns_zone_name = oxide.test found_so_far = 1 version = 2
55632025-03-08T00:51:30.472ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
55642025-03-08T00:51:30.472ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-08 00:40:32.180226 UTC dns_group = external generation = 2
55652025-03-08T00:51:30.472ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 181.231422ms iteration = 12 status = {"generation":2}
55662025-03-08T00:51:30.472ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
55672025-03-08T00:51:30.473ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
55682025-03-08T00:51:30.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55692025-03-08T00:51:30.476ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55702025-03-08T00:51:30.481ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
55712025-03-08T00:51:30.481ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
55722025-03-08T00:51:30.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
55732025-03-08T00:51:30.488ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
55742025-03-08T00:51:30.488ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 195.578526ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
55752025-03-08T00:51:30.493ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55762025-03-08T00:51:30.494ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55772025-03-08T00:51:30.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55782025-03-08T00:51:30.496ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
55792025-03-08T00:51:30.496ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55802025-03-08T00:51:30.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55812025-03-08T00:51:30.498ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
55822025-03-08T00:51:30.498ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
55832025-03-08T00:51:30.498ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 206.175157ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
55842025-03-08T00:51:30.498ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 206.545847ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
55852025-03-08T00:51:30.536ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-08T00:40:26.976319Z, time_modified: 2025-03-08T00:40:26.976319Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5903a422-2123-4b6a-9e2a-1da7d551206b, db_silo: Silo { identity: SiloIdentity { id: 5903a422-2123-4b6a-9e2a-1da7d551206b, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-08T00:40:32.014433Z, time_modified: 2025-03-08T00:40:32.014433Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
55862025-03-08T00:51:30.536ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 245.60084ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]}},"default_endpoint":{"silo_id":"5903a422-2123-4b6a-9e2a-1da7d551206b","tls_certs":["1581ea4ce07f129487a559357598c5e34b57275734a8b97c519aa885d457054f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
55872025-03-08T00:51:30.537ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
55882025-03-08T00:51:30.537ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
55892025-03-08T00:51:30.537ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55902025-03-08T00:51:30.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55912025-03-08T00:51:30.568ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 276.482092ms iteration = 25 status = {"physical_disks_added":0}
55922025-03-08T00:51:30.579ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
55932025-03-08T00:51:30.591ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
55942025-03-08T00:51:30.591ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "160cc97f-2fbd-4968-8828-8a1ef992c419", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55952025-03-08T00:51:30.591ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55962025-03-08T00:51:30.593ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "0c6f6f63-9c10-4150-babc-2832f19079a2", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55972025-03-08T00:51:30.593ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55982025-03-08T00:51:30.595ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
55992025-03-08T00:51:30.595ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 303.729164ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
56002025-03-08T00:51:30.605ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
56012025-03-08T00:51:30.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
56022025-03-08T00:51:30.608ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
56032025-03-08T00:51:30.608ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c77d1d7-af32-4613-92dc-477f3fa15b60", "content-length": "2", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56042025-03-08T00:51:30.610ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
56052025-03-08T00:51:30.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
56062025-03-08T00:51:30.670ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 379.075533ms iteration = 23 status = {}
56072025-03-08T00:51:30.708ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
56082025-03-08T00:51:30.709ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "5af428cc-d131-4317-a625-7d705586e6c1", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
56092025-03-08T00:51:30.709ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
56102025-03-08T00:51:30.711ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "31b4d7db-78c5-4a35-87f8-ef8cd304be7b", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
56112025-03-08T00:51:30.711ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
56122025-03-08T00:51:30.713ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "fea2ca8f-600d-407b-bed9-f55893c83ff9", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
56132025-03-08T00:51:30.713ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
56142025-03-08T00:51:30.713ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
56152025-03-08T00:51:30.713ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
56162025-03-08T00:51:30.714ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "90ba90d8-e9ac-4569-ba27-173ba7c9a090", "content-length": "372", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56172025-03-08T00:51:30.714ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56182025-03-08T00:51:30.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56192025-03-08T00:51:30.715ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56202025-03-08T00:51:30.716ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
56212025-03-08T00:51:30.716ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56222025-03-08T00:51:30.716ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 424.828633ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
56232025-03-08T00:51:30.717ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 425.576511ms iteration = 23 status = null
56242025-03-08T00:51:30.718ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
56252025-03-08T00:51:30.718ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
56262025-03-08T00:51:30.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
56272025-03-08T00:51:30.719ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
56282025-03-08T00:51:30.719ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56292025-03-08T00:51:30.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56302025-03-08T00:51:30.720ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56312025-03-08T00:51:30.721ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7054f21d-f8f9-4761-865d-905a610ae4df dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
56322025-03-08T00:51:30.721ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
56332025-03-08T00:51:30.721ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-08 00:40:30.694594 UTC dns_group = internal generation = 1
56342025-03-08T00:51:30.721ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 430.39287ms iteration = 12 status = {"generation":1}
56352025-03-08T00:51:30.723ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
56362025-03-08T00:51:30.723ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
56372025-03-08T00:51:30.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
56382025-03-08T00:51:30.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56392025-03-08T00:51:30.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56402025-03-08T00:51:30.767ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
56412025-03-08T00:51:30.767ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56422025-03-08T00:51:30.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56432025-03-08T00:51:30.772ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
56442025-03-08T00:51:30.772ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac576d54-60a9-4c9d-8a76-cdae7720c8ae", "content-length": "691", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56452025-03-08T00:51:30.772ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
56462025-03-08T00:51:30.772ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
56472025-03-08T00:51:30.772ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
56482025-03-08T00:51:30.774ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 481.905371ms iteration = 23 status = {}
56492025-03-08T00:51:30.778ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
56502025-03-08T00:51:30.779ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "0101f388-c2f2-4a48-a52b-b3cc3320c09f", "content-length": "134", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56512025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
56522025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
56532025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
56542025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
56552025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56562025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56572025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56582025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
56592025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
56602025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
56612025-03-08T00:51:30.779ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
56622025-03-08T00:51:30.779ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56632025-03-08T00:51:30.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56642025-03-08T00:51:30.782ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
56652025-03-08T00:51:30.782ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a1f42e4-afbc-42c4-91ee-65a4137976e9", "content-length": "2", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56662025-03-08T00:51:30.782ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
56672025-03-08T00:51:30.782ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac61bbf0-fa7a-4836-a7c3-3b28c5d79a34", "content-length": "52", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56682025-03-08T00:51:30.782ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
56692025-03-08T00:51:30.782ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
56702025-03-08T00:51:30.792ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
56712025-03-08T00:51:30.792ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
56722025-03-08T00:51:30.897ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "bda38eea-8a52-45b5-b4f3-fdfa4da239d8", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56732025-03-08T00:51:30.897ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
56742025-03-08T00:51:30.897ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
56752025-03-08T00:51:30.897ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b8b8453-3b91-4bf8-8630-07de7417909d", "content-length": "682", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56762025-03-08T00:51:30.898ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56772025-03-08T00:51:30.898ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56782025-03-08T00:51:30.898ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "aae4018d-f6d9-4da5-9658-528b2119e274", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56792025-03-08T00:51:30.898ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56802025-03-08T00:51:30.898ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56812025-03-08T00:51:30.900ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56822025-03-08T00:51:30.900ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56832025-03-08T00:51:30.902ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56842025-03-08T00:51:30.902ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56852025-03-08T00:51:30.903ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56862025-03-08T00:51:30.903ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56872025-03-08T00:51:30.906ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56882025-03-08T00:51:30.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56892025-03-08T00:51:30.907ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
56902025-03-08T00:51:30.907ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
56912025-03-08T00:51:30.907ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "15b12957-4f57-40dc-8577-022dcb6263c1", "date": "Sat, 08 Mar 2025 00:51:30 GMT"} })
56922025-03-08T00:51:30.908ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56932025-03-08T00:51:30.908ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56942025-03-08T00:51:30.909ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 617.22634ms iteration = 23 status = {}
56952025-03-08T00:51:30.914ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 622.448389ms iteration = 23 status = {}
56962025-03-08T00:51:31.039ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
56972025-03-08T00:51:31.039ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::a]:12221
56982025-03-08T00:51:31.069ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::a]:12221/metrics/producers
56992025-03-08T00:51:31.069ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::a]:40878
57002025-03-08T00:51:31.069ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:40878 req_id = a0ccb397-a109-4030-ba12-a9b4cdf1abb5 roles = RoleSet { roles: {} } uri = /metrics/producers
57012025-03-08T00:51:31.070ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:40878 req_id = a0ccb397-a109-4030-ba12-a9b4cdf1abb5 resource = Database result = Ok(()) uri = /metrics/producers
57022025-03-08T00:51:31.111ZINFOnexus: assigned collector to new producer collector_id = c5a56912-50aa-493d-9612-f426cf7f1072 file = nexus/src/app/oximeter.rs:76 producer_id = 841ee309-4e1e-443f-ae06-2ad5eb454a8d
57032025-03-08T00:51:31.111ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 41748 local_addr = [fd00:1122:3344:101::a]:12221 method = POST remote_addr = [fd00:1122:3344:101::a]:40878 req_id = a0ccb397-a109-4030-ba12-a9b4cdf1abb5 response_code = 201 uri = /metrics/producers
57042025-03-08T00:51:31.111ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "a0ccb397-a109-4030-ba12-a9b4cdf1abb5", "content-length": "41", "date": "Sat, 08 Mar 2025 00:51:31 GMT"} })
57052025-03-08T00:51:31.112ZDEBGnexus (producer-registration-task): registered with nexus successfully
57062025-03-08T00:51:31.112ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
57072025-03-08T00:51:32.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 372 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 39d30bbb-0644-4480-9877-d9ddd1fe94e0 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
57082025-03-08T00:51:40.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
57092025-03-08T00:51:40.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57102025-03-08T00:51:40.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57112025-03-08T00:51:40.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
57122025-03-08T00:51:40.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57132025-03-08T00:51:40.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57142025-03-08T00:51:40.294ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57152025-03-08T00:51:40.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57162025-03-08T00:51:40.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57172025-03-08T00:51:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
57182025-03-08T00:51:40.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57192025-03-08T00:51:40.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57202025-03-08T00:51:40.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57212025-03-08T00:51:40.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57222025-03-08T00:51:40.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57232025-03-08T00:51:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
57242025-03-08T00:51:40.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57252025-03-08T00:51:40.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57262025-03-08T00:51:40.315ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
57272025-03-08T00:51:40.315ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.202947ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
57282025-03-08T00:51:40.324ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.82326ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57292025-03-08T00:51:42.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 335 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 49466fd2-2757-460f-939a-77c7d2dc4df9 response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
57302025-03-08T00:51:50.291ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
57312025-03-08T00:51:50.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
57322025-03-08T00:51:50.291ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57332025-03-08T00:51:50.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57342025-03-08T00:51:50.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57352025-03-08T00:51:50.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57362025-03-08T00:51:50.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57372025-03-08T00:51:50.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57382025-03-08T00:51:50.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57392025-03-08T00:51:50.296ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.513291ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57402025-03-08T00:51:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
57412025-03-08T00:51:50.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57422025-03-08T00:51:50.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57432025-03-08T00:51:50.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57442025-03-08T00:51:50.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57452025-03-08T00:51:50.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57462025-03-08T00:51:50.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
57472025-03-08T00:51:50.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57482025-03-08T00:51:50.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57492025-03-08T00:51:50.313ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
57502025-03-08T00:51:50.313ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.47362ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
57512025-03-08T00:51:52.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 379 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = f9fd68e4-dcab-4235-b9d1-b6e0222be3be response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
57522025-03-08T00:51:57.639ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::a]:12221 remote_addr = [fd00:1122:3344:101::d]:45523
57532025-03-08T00:51:57.639ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45523 req_id = b5c34b02-3348-4a0f-8ad4-3c47a5e33c3a roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
57542025-03-08T00:51:57.639ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45523 req_id = b5c34b02-3348-4a0f-8ad4-3c47a5e33c3a resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
57552025-03-08T00:51:57.642ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3423 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45523 req_id = b5c34b02-3348-4a0f-8ad4-3c47a5e33c3a response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?sort_by=id_ascending
57562025-03-08T00:51:57.643ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45523 req_id = c8f4340f-f27e-4743-90a0-50817cd1723f roles = RoleSet { roles: {} } uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
57572025-03-08T00:51:57.643ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45523 req_id = c8f4340f-f27e-4743-90a0-50817cd1723f resource = Database result = Ok(()) uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
57582025-03-08T00:51:57.645ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2136 local_addr = [fd00:1122:3344:101::a]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:45523 req_id = c8f4340f-f27e-4743-90a0-50817cd1723f response_code = 200 uri = /metrics/collectors/c5a56912-50aa-493d-9612-f426cf7f1072/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkOWZhNThkOS1kMjgzLTQ1MTAtODJkZS1iNzAxMzY3NTdiOGYifX0%3D
57592025-03-08T00:52:00.290ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
57602025-03-08T00:52:00.290ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
57612025-03-08T00:52:00.290ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
57622025-03-08T00:52:00.290ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
57632025-03-08T00:52:00.290ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
57642025-03-08T00:52:00.290ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
57652025-03-08T00:52:00.290ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57662025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57672025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
57682025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
57692025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
57702025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
57712025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
57722025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57732025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
57742025-03-08T00:52:00.291ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57752025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
57762025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
57772025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57782025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
57792025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
57802025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
57812025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
57822025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57832025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
57842025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
57852025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
57862025-03-08T00:52:00.292ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
57872025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57882025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
57892025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
57902025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
57912025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
57922025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
57932025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
57942025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57952025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
57962025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
57972025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
57982025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
57992025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
58002025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
58012025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58022025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
58032025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
58042025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
58052025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
58062025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
58072025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
58082025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58092025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
58102025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58112025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
58122025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
58132025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
58142025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
58152025-03-08T00:52:00.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
58162025-03-08T00:52:00.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58172025-03-08T00:52:00.294ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.295742ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
58182025-03-08T00:52:00.294ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1603, tv_nsec: 319059359 } })
58192025-03-08T00:52:00.294ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
58202025-03-08T00:52:00.294ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58212025-03-08T00:52:00.295ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1603, tv_nsec: 319509770 } })
58222025-03-08T00:52:00.295ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
58232025-03-08T00:52:00.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58242025-03-08T00:52:00.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58252025-03-08T00:52:00.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58262025-03-08T00:52:00.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
58272025-03-08T00:52:00.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
58282025-03-08T00:52:00.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
58292025-03-08T00:52:00.298ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58302025-03-08T00:52:00.300ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58312025-03-08T00:52:00.300ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58322025-03-08T00:52:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
58332025-03-08T00:52:00.300ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
58342025-03-08T00:52:00.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
58352025-03-08T00:52:00.302ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58362025-03-08T00:52:00.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
58372025-03-08T00:52:00.303ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
58382025-03-08T00:52:00.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
58392025-03-08T00:52:00.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
58402025-03-08T00:52:00.304ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
58412025-03-08T00:52:00.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
58422025-03-08T00:52:00.305ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
58432025-03-08T00:52:00.305ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58442025-03-08T00:52:00.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58452025-03-08T00:52:00.311ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
58462025-03-08T00:52:00.311ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58472025-03-08T00:52:00.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
58482025-03-08T00:52:00.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58492025-03-08T00:52:00.312ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-87826585-0843-40d2-a3ce-382fa54dbc5c.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1603, tv_nsec: 336887969 } })
58502025-03-08T00:52:00.312ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
58512025-03-08T00:52:00.313ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
58522025-03-08T00:52:00.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
58532025-03-08T00:52:00.316ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58542025-03-08T00:52:00.317ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58552025-03-08T00:52:00.317ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58562025-03-08T00:52:00.317ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
58572025-03-08T00:52:00.317ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 25.847098ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
58582025-03-08T00:52:00.318ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58592025-03-08T00:52:00.320ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
58602025-03-08T00:52:00.320ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
58612025-03-08T00:52:00.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
58622025-03-08T00:52:00.321ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
58632025-03-08T00:52:00.321ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
58642025-03-08T00:52:00.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
58652025-03-08T00:52:00.321ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
58662025-03-08T00:52:00.321ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
58672025-03-08T00:52:00.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
58682025-03-08T00:52:00.323ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
58692025-03-08T00:52:00.323ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58702025-03-08T00:52:00.323ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58712025-03-08T00:52:00.347ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
58722025-03-08T00:52:00.347ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
58732025-03-08T00:52:00.347ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 55.654433ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
58742025-03-08T00:52:00.348ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
58752025-03-08T00:52:00.348ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.464429ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
58762025-03-08T00:52:00.349ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58772025-03-08T00:52:00.352ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
58782025-03-08T00:52:00.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
58792025-03-08T00:52:00.353ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
58802025-03-08T00:52:00.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
58812025-03-08T00:52:00.354ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
58822025-03-08T00:52:00.354ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
58832025-03-08T00:52:00.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
58842025-03-08T00:52:00.357ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 65.226009ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
58852025-03-08T00:52:00.361ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58862025-03-08T00:52:00.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58872025-03-08T00:52:00.365ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 73.514844ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
58882025-03-08T00:52:00.377ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
58892025-03-08T00:52:00.377ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
58902025-03-08T00:52:00.377ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
58912025-03-08T00:52:00.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
58922025-03-08T00:52:00.378ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
58932025-03-08T00:52:00.378ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 86.69977ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
58942025-03-08T00:52:00.419ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
58952025-03-08T00:52:00.419ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
58962025-03-08T00:52:00.419ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bc48d61-e652-44a0-9b3e-18b72446e58b", "content-length": "26", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
58972025-03-08T00:52:00.419ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
58982025-03-08T00:52:00.421ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
58992025-03-08T00:52:00.450ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 158.112161ms iteration = 26 status = {"physical_disks_added":0}
59002025-03-08T00:52:00.451ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59012025-03-08T00:52:00.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59022025-03-08T00:52:00.462ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
59032025-03-08T00:52:00.462ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
59042025-03-08T00:52:00.463ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "87afc3b9-e4a5-4e04-bf37-e83f68a918bf", "content-length": "372", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59052025-03-08T00:52:00.463ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59062025-03-08T00:52:00.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59072025-03-08T00:52:00.467ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59082025-03-08T00:52:00.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5909 Batch: [InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1024, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1025, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1026, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1027, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1028, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1029, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1030, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1031, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1032, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1033, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }, InvNvmeDiskFirmware { inv_collection_id: 057bd338-e1ab-4773-96b6-de3ce7964654 (collection), sled_id: 87826585-0843-40d2-a3ce-382fa54dbc5c (sled), slot: 1034, number_of_slots: SqlU8(1), active_slot: SqlU8(1), next_active_slot: None, slot1_is_read_only: true, slot_firmware_versions: [Some("SYNTH1")] }]
59102025-03-08T00:52:00.493ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59112025-03-08T00:52:00.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59122025-03-08T00:52:00.495ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59132025-03-08T00:52:00.495ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59142025-03-08T00:52:00.495ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
59152025-03-08T00:52:00.495ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
59162025-03-08T00:52:00.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59172025-03-08T00:52:00.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
59182025-03-08T00:52:00.496ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "186d301c-3145-49d2-8d83-a79eaa6ddbfd", "content-length": "26", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59192025-03-08T00:52:00.496ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
59202025-03-08T00:52:00.498ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 206.071835ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
59212025-03-08T00:52:00.498ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
59222025-03-08T00:52:00.498ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 zone_address = fd00:1122:3344:101::2
59232025-03-08T00:52:00.498ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
59242025-03-08T00:52:00.498ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aef60894-aa67-4ed9-8449-9e03c337f65e", "content-length": "26", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59252025-03-08T00:52:00.498ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
59262025-03-08T00:52:00.499ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 207.419748ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
59272025-03-08T00:52:00.500ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 208.079775ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
59282025-03-08T00:52:00.500ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
59292025-03-08T00:52:00.500ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
59302025-03-08T00:52:00.500ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59312025-03-08T00:52:00.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
59322025-03-08T00:52:00.500ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
59332025-03-08T00:52:00.508ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
59342025-03-08T00:52:00.508ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
59352025-03-08T00:52:00.508ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2e67a5b1-71f0-454c-991e-1c218276fc90", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59362025-03-08T00:52:00.508ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59372025-03-08T00:52:00.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59382025-03-08T00:52:00.511ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59392025-03-08T00:52:00.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59402025-03-08T00:52:00.514ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59412025-03-08T00:52:00.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59422025-03-08T00:52:00.518ZINFOnexus (BackgroundTasks): VPC route manager sled 87826585-0843-40d2-a3ce-382fa54dbc5c push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
59432025-03-08T00:52:00.518ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
59442025-03-08T00:52:00.518ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "1c1bffb5-de67-4f78-bb2f-e0d826d54062", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59452025-03-08T00:52:00.520ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 228.069417ms iteration = 24 status = {}
59462025-03-08T00:52:00.522ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59472025-03-08T00:52:00.525ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 87826585-0843-40d2-a3ce-382fa54dbc5c, lookup_type: ById(87826585-0843-40d2-a3ce-382fa54dbc5c) } result = Ok(())
59482025-03-08T00:52:00.525ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59492025-03-08T00:52:00.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59502025-03-08T00:52:00.525ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 235.04668ms iteration = 24 status = {}
59512025-03-08T00:52:00.526ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 234.740972ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
59522025-03-08T00:52:00.527ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 235.723986ms iteration = 24 status = 0
59532025-03-08T00:52:00.532ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
59542025-03-08T00:52:00.532ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
59552025-03-08T00:52:00.532ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 240.366391ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
59562025-03-08T00:52:00.551ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
59572025-03-08T00:52:00.551ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5077afd-e35f-4357-b5a1-07cbf75b9132", "content-length": "2", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59582025-03-08T00:52:00.553ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
59592025-03-08T00:52:00.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
59602025-03-08T00:52:00.556ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 265.272695ms iteration = 24 status = null
59612025-03-08T00:52:00.576ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59622025-03-08T00:52:00.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59632025-03-08T00:52:00.578ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59642025-03-08T00:52:00.580ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
59652025-03-08T00:52:00.580ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59662025-03-08T00:52:00.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59672025-03-08T00:52:00.581ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59682025-03-08T00:52:00.583ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
59692025-03-08T00:52:00.583ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59702025-03-08T00:52:00.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59712025-03-08T00:52:00.609ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59722025-03-08T00:52:00.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59732025-03-08T00:52:00.611ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: ca4a7b6e-327d-40f9-a885-12eabbb4e3d4, rack_id: 533106a5-4f1c-4a48-aa36-2a30759fbe03, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(61628d69-5fe0-495b-8add-dfbccef57c71) } rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
59742025-03-08T00:52:00.611ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59752025-03-08T00:52:00.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59762025-03-08T00:52:00.621ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
59772025-03-08T00:52:00.621ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "f48757d5-af23-40ce-847a-d53c2d1059b0", "content-length": "134", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59782025-03-08T00:52:00.621ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
59792025-03-08T00:52:00.621ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 61628d69-5fe0-495b-8add-dfbccef57c71
59802025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
59812025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
59822025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
59832025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
59842025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
59852025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
59862025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
59872025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
59882025-03-08T00:52:00.622ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03
59892025-03-08T00:52:00.622ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59902025-03-08T00:52:00.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59912025-03-08T00:52:00.624ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
59922025-03-08T00:52:00.624ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "1acc7631-9c21-4738-bfd9-abf980b6aa83", "content-length": "2", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59932025-03-08T00:52:00.624ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
59942025-03-08T00:52:00.624ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa48dc87-e61b-4b8b-aafa-0a046d4bdec6", "content-length": "52", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59952025-03-08T00:52:00.624ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch_location = Switch0
59962025-03-08T00:52:00.624ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
59972025-03-08T00:52:00.689ZDEBGnexus (BackgroundTasks): client request SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
59982025-03-08T00:52:00.748ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0fd31f2e-a000-456a-b0a2-b741a023fa3e", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
59992025-03-08T00:52:00.748ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 switch = Switch0
60002025-03-08T00:52:00.748ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
60012025-03-08T00:52:00.752ZDEBGnexus (BackgroundTasks): client response SledAgent = 87826585-0843-40d2-a3ce-382fa54dbc5c background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5a9a5e0-386f-4f39-9b8b-46118af2cca5", "content-length": "691", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
60022025-03-08T00:52:00.752ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
60032025-03-08T00:52:00.752ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
60042025-03-08T00:52:00.752ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
60052025-03-08T00:52:00.752ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 533106a5-4f1c-4a48-aa36-2a30759fbe03 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "77ac5b89-5666-44ba-9a62-210390b92bcb", "content-length": "682", "date": "Sat, 08 Mar 2025 00:52:00 GMT"} })
60062025-03-08T00:52:00.752ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60072025-03-08T00:52:00.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60082025-03-08T00:52:00.754ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 462.725898ms iteration = 24 status = {}
60092025-03-08T00:52:00.757ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60102025-03-08T00:52:00.757ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60112025-03-08T00:52:00.761ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60122025-03-08T00:52:00.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60132025-03-08T00:52:00.763ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60142025-03-08T00:52:00.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60152025-03-08T00:52:00.771ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 479.508502ms iteration = 24 status = {}
60162025-03-08T00:52:02.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 330 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = 0222df7e-fc61-4923-889e-90bf934ae8ac response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d
60172025-03-08T00:52:10.291ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
60182025-03-08T00:52:10.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60192025-03-08T00:52:10.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60202025-03-08T00:52:10.291ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
60212025-03-08T00:52:10.291ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
60222025-03-08T00:52:10.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60232025-03-08T00:52:10.293ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60242025-03-08T00:52:10.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
60252025-03-08T00:52:10.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60262025-03-08T00:52:10.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60272025-03-08T00:52:10.298ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60282025-03-08T00:52:10.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60292025-03-08T00:52:10.300ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60302025-03-08T00:52:10.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: a346f089-8033-4b92-b854-361d1da5cab2, lookup_type: ById(a346f089-8033-4b92-b854-361d1da5cab2) } result = Ok(())
60312025-03-08T00:52:10.303ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60322025-03-08T00:52:10.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60332025-03-08T00:52:10.317ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = a346f089-8033-4b92-b854-361d1da5cab2 original_time_created = 2025-03-08 00:40:23.114725 UTC target_id = a346f089-8033-4b92-b854-361d1da5cab2 time_created = 2025-03-08 00:40:23.114725 UTC
60342025-03-08T00:52:10.317ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.247298ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"a346f089-8033-4b92-b854-361d1da5cab2","time_created":"2025-03-08T00:40:23.114725Z"}
60352025-03-08T00:52:10.322ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
60362025-03-08T00:52:10.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60372025-03-08T00:52:10.360ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.553281ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
60382025-03-08T00:52:12.649ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 370 local_addr = [fd00:1122:3344:101::a]:52973 method = GET remote_addr = [fd00:1122:3344:101::d]:54785 req_id = e641db4d-7ad3-4fcd-9e3b-c896befa348f response_code = 200 uri = /841ee309-4e1e-443f-ae06-2ad5eb454a8d